seafile-pro-server-3.1.6 library creation not working

[egroeper]

Hi, this could be a configuration issue, but I can't find anything. I'm running seafile-pro 3.1.6 with ceph backend, cluster setup, mysql, nginx (HTTPS) and of course memcached. At this step I don't use a load balancer and only have a single seafile-node. After logging in I try to create a library. The Web-UI window doesn't show any reaction after clicking submit. With web debugging I see a POST request to https://<cluster-url>/ajax/repo/create/. After some time this request ends with a 504 gateway timeout (according to web debugger. The Web-UI window still doesn't show any change).

At this point I'm following the logs with tail -f to see when something happens. During the request I don't see anything in the logs (seafile.log, seahub_django_request.log, ccnet.log, fileserver.log).

By chance I found out, that I get some interesting (and useful?) information when stopping seahub:

./seahub.sh stop

Stopping seahub ...
2014-09-18 15:02:17,327 [ERROR] django.request:212 handle_uncaught_exception Internal Server Error: /ajax/repo/create/
Traceback (most recent call last):
  File "/data/seafile/seafile-pro-server-3.1.6/seahub/thirdpart/Django-1.5.1-py2.6.egg/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/data/seafile/seafile-pro-server-3.1.6/seahub/seahub/auth/decorators.py", line 69, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/data/seafile/seafile-pro-server-3.1.6/seahub/seahub/views/ajax.py", line 1704, in repo_create
    uuid, magic_str, encrypted_file_key)
  File "/data/seafile/seafile-pro-server-3.1.6/seahub/seahub/views/ajax.py", line 1655, in _create_repo_common
    username, None)
  File "/data/seafile/seafile-pro-server-3.1.6/seafile/lib64/python2.6/site-packages/seaserv/api.py", line 47, in create_repo
    return seafserv_threaded_rpc.create_repo(name, desc, username, passwd)
  File "/data/seafile/seafile-pro-server-3.1.6/seafile/lib64/python2.6/site-packages/pysearpc/client.py", line 110, in newfunc
    ret_str = self.call_remote_func_sync(fcall_str)
  File "/data/seafile/seafile-pro-server-3.1.6/seafile/lib64/python2.6/site-packages/ccnet/rpc.py", line 78, in call_remote_func_sync
    ret = self._real_call(client, req_id, fcall_str)
  File "/data/seafile/seafile-pro-server-3.1.6/seafile/lib64/python2.6/site-packages/ccnet/rpc.py", line 42, in _real_call
    rsp = client.read_response()
  File "/data/seafile/seafile-pro-server-3.1.6/seafile/lib64/python2.6/site-packages/ccnet/sync_client.py", line 29, in read_response
    packet = read_packet(self._connfd)
  File "/data/seafile/seafile-pro-server-3.1.6/seafile/lib64/python2.6/site-packages/ccnet/packet.py", line 108, in read_packet
    hdr = recvall(fd, CCNET_HEADER_LENGTH)
  File "/data/seafile/seafile-pro-server-3.1.6/seafile/lib64/python2.6/site-packages/ccnet/utils.py", line 11, in recvall
    new = fd.recv(remain)
KeyboardInterrupt
Done.

[09/18/14 07:02:17] ../common/peer.c(943): Local peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor service-proxy-proc(-1002) for bad update: 515 peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor service-stub-proc(1014) for bad update: 515 peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor threaded-rpcserver-proc(-1001) for bad update: 515 peer down
[09/18/14 07:02:17] ccnet_processor_handle_update: [Proc] Shutdown processor threaded-rpcserver-proc(-1014) for bad update: 515 peer down
[09/18/14 07:02:17] ../common/peer.c(943): Local peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor service-proxy-proc(-1002) for bad update: 515 peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor service-stub-proc(1012) for bad update: 515 peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor threaded-rpcserver-proc(-1001) for bad update: 515 peer down
[09/18/14 07:02:17] ccnet_processor_handle_update: [Proc] Shutdown processor threaded-rpcserver-proc(-1012) for bad update: 515 peer down
[09/18/14 07:02:17] ../common/peer.c(943): Local peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor threaded-rpcserver-proc(-1001) for bad update: 515 peer down
[09/18/14 07:02:17] ../common/peer.c(943): Local peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor service-proxy-proc(-1002) for bad update: 515 peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor service-stub-proc(1013) for bad update: 515 peer down
[09/18/14 07:02:17] ../common/processor.c(218): [Proc] Shutdown processor threaded-rpcserver-proc(-1001) for bad update: 515 peer down
[09/18/14 07:02:17] ccnet_processor_handle_update: [Proc] Shutdown processor threaded-rpcserver-proc(-1013) for bad update: 515 peer down

To me it seems like the problem is somewhere in the communication between seahub and another component (seaf-server?). If I try to create the library twice, I get two exceptions when stopping seahub.

If I remember correctly, a very similar setup worked fine using seafile-pro 3.1.1.

[egroeper]

I tried to setup a debugging/testing environment, but that didn't get me any further:

cd /data/seafile/seafile-server-latest
LD_LIBRARY_PATH=./seafile/lib/:./seafile/lib64:${LD_LIBRARY_PATH} ./seafile/bin/ccnet-server -c /data/seafile/ccnet -f /data/seafile/logs/ccnet_debug.log -d -M 50 -P /data/seafile/pids/ccnet.pid -D all
LD_LIBRARY_PATH=./seafile/lib/:./seafile/lib64:${LD_LIBRARY_PATH} ./seafile/bin/seaf-server -c /data/seafile/ccnet -d /data/seafile/seafile-data -l /data/seafile/logs/seafile_debug.log -P /data/seafile/pids/seaf-server.pid -D all
LD_LIBRARY_PATH=./seafile/lib/:./seafile/lib64:${LD_LIBRARY_PATH} ./seafile/bin/fileserver -c /data/seafile/ccnet -d /data/seafile/seafile-data -l /data/seafile/logs/fileserver_debug.log -P /data/seafile/pids/fileserver.pid -D all

export INSTALLPATH=`pwd`
export PYTHONPATH=${INSTALLPATH}/seafile/lib/python2.6/site-packages:${INSTALLPATH}/seafile/lib64/python2.6/site-packages:${INSTALLPATH}/seahub/thirdpart:$PYTHONPATH
export PYTHONPATH=${INSTALLPATH}/seafile/lib/python2.7/site-packages:${INSTALLPATH}/seafile/lib64/python2.7/site-packages:$PYTHONPATH
export PYTHONPATH=$PYTHONPATH:${INSTALLPATH}/pro/python
export PYTHONPATH=$PYTHONPATH:${INSTALLPATH}/seahub-extra/
export PYTHONPATH=$PYTHONPATH:${INSTALLPATH}/seahub-extra/thirdparts
export CCNET_CONF_DIR=/data/seafile/ccnet/
export SEAFILE_CONF_DIR=/data/seafile/seafile-data
/usr/bin/python2.7 -m seafevents.main --config-file /data/seafile/pro-data/seafevents.conf --loglevel debug --logfile /data/seafile/logs/seafevents.log -P /data/seafile/pids/seafevents.pid &

./seahub.sh start-fastcgi

[egroeper]

If I remember correctly, a very similar setup worked fine using seafile-pro 3.1.1.

I now downgraded to 3.1.1 (since there are no db schema changes, it's no problem) and can confirm, that in 3.1.1 it's no problem to create libraries.

[freeplant]

Can you try the v3.1.7 beta?

[egroeper]

v3.1.7 beta works.

[egroeper]

@freeplant: This issue seems to be back. At least I got it with seafile-pro-4.2.1 and seafile-pro.4.1.1. Some regression? Or do you have a hint which config issues could cause this? When does this happen? ccnet runs locally on each node, doesn't it? So this is a problem when seahub want to communicate with ccnet on the same node?

[freeplant]

It seems to be a problem that the Seafile server spend too much time in creating a library. Maybe this is related to the Ceph backend?

[egroeper]

@freeplant You are right. I found the problem. There was a problem with our firewall. The setup changed and I forgot to update the settings at one place. Now it works fine. Sorry.

But it makes me wonder, that there weren't any error messages in seafile.log (or ccnet.log?). I'm sure there have been proper error messages when such things happened during my tests in the past. Because there weren't any now, I erroneously excepted the connection to the ceph cluster as root cause. Thinking about that: Perhaps I'm confusing that with ceph authentication issues I had in the past (non admin user).

Anyway: I'm happy that it works now. Perhaps you could guard against (my) stupidity with proper error messages in the future. ;-)

[killing]

@egroeper The error messages are still there. The error can't be detected until the connection times out since it's blocked by firewall.

[egroeper]

@killing I can't confirm that. After stopping seahub.sh the KeyboardInterrupt message from above appears in the logs. I can't find any "[Block backend] Cannot connect to cluster" messages. The cluster ran for several hours. That should be enough for timeouts, shouldn't it? Perhaps they are somehow hidden now (don't get executed because of the system waiting for other stuff)?

[egroeper]

This message seems to be generated in seaf-server. Could it be, that it doesn't get executed on a fresh install with no libraries/repositories, yet? Perhaps it only gets executed when trying to access data and not when trying to create a repository? I would have a look at it myself, but that part of the code is proprietary, afaik.