hang during syncs of large collections

[tomtor]

After running about an hour gsync always ends hanging in a random transfer.

When running in debug mode and after manual interrupt:

DEBUG: drive/:708:update(): * uploading next chunk... DEBUG: drive/:708:update(): * uploading next chunk... DEBUG: drive/:708:update(): * uploading next chunk... DEBUG: drive/:708:update(): * uploading next chunk...

^CDEBUG: sync/:200:_sync(): Interrupted

Interrupted close failed in file object destructor: sys.excepthook is missing lost sys.stderr tom@swan:/media/scratch/backup/gdrive$

[tomtor]

On the current master with some minor changes to get it compiled I get the following stacktrace:

f+++++++++ 2007/maastricht-07/imgp0534.pef 10485760 86% 155.15KB/s 0:01:06^C Interrupted sent 2797971312 bytes received 0 bytes 778014.47 bytes/sec DEBUG: KeyboardInterrupt(): File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/EGG-INFO/scripts/gsync", line 82, in main Crawler(src, dest).run() File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/crawler.py", line 193, in run self._walk(srcpath, self._walk_callback, self._dev) File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/crawler.py", line 163, in _walk self._sync(absfile) File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/init.py", line 263, in call self._sync(path) File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/init.py", line 315, in _sync self.dst.create(dst_path, src_file) File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/file/init.py", line 443, in create self.create_file(path, src_obj) File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/file/__init.py", line 350, in create_file self._update_data(path, src) File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/file/remote/__init.py", line 132, in _update_data progress_callback=progress File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/drive/init.py", line 874, in update status, res = req.next_chunk() File "build/bdist.linux-x86_64/egg/oauth2client/util.py", line 132, in positional_wrapper return wrapped(_args, _kwargs) File "build/bdist.linux-x86_64/egg/apiclient/http.py", line 867, in next_chunk headers=headers) File "build/bdist.linux-x86_64/egg/oauth2client/util.py", line 132, in positional_wrapper return wrapped(_args, _kwargs) File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 497, in new_request redirections, connection_type) File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1569, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1316, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1285, in _conn_request response = conn.getresponse() File "/usr/lib/python2.7/httplib.py", line 1045, in getresponse response.begin() File "/usr/lib/python2.7/httplib.py", line 409, in begin version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 365, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib/python2.7/socket.py", line 476, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/python2.7/ssl.py", line 341, in recv return self.read(buflen) File "/usr/lib/python2.7/ssl.py", line 260, in read return self._sslobj.read(len)

[tomtor]

I made a dirty hack which got me uploading through the night:

diff --git a/libgsync/drive/__init__.py b/libgsync/drive/__init__.py
index 7ea043e..9da03e8 100644
--- a/libgsync/drive/__init__.py
+++ b/libgsync/drive/__init__.py
@@ -6,6 +6,12 @@
 """The GSync Drive module that provides an interface to the Google Drive"""

 import os, sys, re, datetime, shelve, time
+import signal
+
+def handler(signum, frame):
+    print 'Signal handler called with signal', signum
+
+signal.signal(signal.SIGALRM, handler)

 from contextlib import contextmanager

@@ -871,7 +877,9 @@ class Drive(object):
                     while res is None:
                         debug(" * uploading next chunk...")

+                        signal.alarm(10)
                         status, res = req.next_chunk()
+                        signal.alarm(0) # Disable the signal
                         if status:
                             progress_callback(status)

[mullenkamp]

I have the same problem...but I don't have a dirty hack... :(

[fnkr]

+1