search

iwonbigbro / gsync

RSync for Google Drive - GSync
Other
238 stars 50 forks source link

Error: IOError(36, 'File name too long') when uploading content of new directory #38

Open lionandoil opened 10 years ago

lionandoil commented 10 years ago

Hello,

when I try to upload a new directory with content it successfully creates the top directory on Drive but then fails with an IOError when uploading/creating the actual content, whether that content is a directory or file. Reproducable in the following ways, error message below:

mkdir foo
touch foo/bar # alternatively mkdir foo/bar, makes no difference
gsync -r foo drive:/// # error
gsync -r foo/ drive:///foo # error
cd foo & gsync -r . drive:///foo # error

I get the following output:

DEBUG:  /usr/local/bin/gsync:25:<module>(): {'--8-bit-output': 0,
 '--append': 0,
 '--backup': 0,
 '--backup-dir': [],
 '--bwlimit': [],
 '--checksum': 0,
 '--chmod': [],
 '--compare-dest': [],
 '--contimeout': [],
 '--copy-dest': [],
 '--copy-dirlinks': 0,
 '--copy-links': 0,
 '--copy-unsafe-links': 0,
 '--cvs-exclude': 0,
 '--debug': 1,
 '--del': 0,
 '--delay-updates': 0,
 '--delete': 0,
 '--delete-after': 0,
 '--delete-before': 0,
 '--delete-delay': 0,
 '--delete-during': 0,
 '--delete-excluded': 0,
 '--dirs': 0,
 '--dry-run': 0,
 '--exclude': [],
 '--exclude-from': [],
 '--executability': 0,
 '--existing': 0,
 '--fake-super': 0,
 '--files-from': [],
 '--filter': [],
 '--force': 0,
 '--from0': 0,
 '--fuzzy': 0,
 '--group': 0,
 '--hard-links': 0,
 '--help': 0,
 '--human-readable': 0,
 '--ignore-errors': 0,
 '--ignore-existing': 0,
 '--ignore-times': 0,
 '--include': [],
 '--include-from': [],
 '--itemize-changes': 1,
 '--keep-dirlinks': 0,
 '--links': 0,
 '--list-only': 0,
 '--log-file': [],
 '--log-file-format': [],
 '--max-delete': [],
 '--max-size': [],
 '--min-size': [],
 '--modify-window': [],
 '--no-implied-dirs': 0,
 '--omit-dir-times': 0,
 '--one-file-system': 0,
 '--out-format': [],
 '--owner': 0,
 '--partial': 0,
 '--partial-dir': [],
 '--perms': 0,
 '--progress': 0,
 '--protect-args': 0,
 '--proxy': 0,
 '--prune-empty-dirs': 0,
 '--quiet': 0,
 '--recursive': 1,
 '--relative': 0,
 '--remove-source-files': 0,
 '--safe-links': 0,
 '--size-only': 0,
 '--sparse': 0,
 '--stats': 0,
 '--suffix': [],
 '--super': 0,
 '--temp-dir': [],
 '--timeout': [],
 '--times': 0,
 '--update': 0,
 '--verbose': 0,
 '--version': 0,
 '-F': 0,
 '-P': 0}
DEBUG:  /usr/local/bin/gsync:54:<module>(): Creating crawler for: 'foo'
DEBUG:     drive/:174:__init__(): Initialising drive
DEBUG:     drive/:191:__init__(): Initialisation complete
DEBUG:    drive/:462:stat(): Checking pcache for path: 'drive://'
DEBUG:    drive/:480:stat(): Path is root: 'drive://'
DEBUG:   crawler.py:61:__init__(): Dest is a directory, not forcing dest file: 'drive://'
DEBUG:   crawler.py:78:__init__(): Dest has trailing slash, not forcing dest file: drive:///
DEBUG:   crawler.py:84:__init__(): force_dest_file = False
DEBUG:   crawler.py:137:run(): Source srcpath: 'foo'
DEBUG:   crawler.py:138:run(): Source basepath: ''
DEBUG:   crawler.py:139:run(): Source path: 'foo'
DEBUG:     sync/file/factory.py:10:create(): SyncFileFactory.create('')
DEBUG:     sync/file/factory.py:25:create(): Creating SyncFileLocal('.')
DEBUG:     sync/file/factory.py:10:create(): SyncFileFactory.create('drive:///')
DEBUG:     sync/file/factory.py:17:create(): Creating SyncFileRemote('drive://')
DEBUG:   crawler.py:147:run(): Enumerating: 'foo'
DEBUG:    crawler.py:102:_walk(): Walking: 'foo'
DEBUG:    crawler.py:112:_walk(): Synchronising directory: 'foo'
DEBUG:      sync/:34:_sync(): Synchronising: 'foo'
DEBUG:       sync/file/:338:relativeTo(): Creating relative path from '^./+' and 'foo'
DEBUG:      sync/:37:_sync(): Destination: SyncFileRemote('drive://')
DEBUG:      sync/:38:_sync(): Relative: 'foo'
DEBUG:        sync/file/:193:getPath(): Joining: '.' with 'foo'
DEBUG:       sync/file/local/:30:getInfo(): Fetching local file metadata: './foo'
DEBUG:       sync/file/local/:59:getInfo(): Local mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 11, 33, 12, 969249, tzinfo=tzutc()))
DEBUG:       sync/file/local/:64:getInfo(): Local mtime: 2013-11-30T11:33:12.969249+00:00
DEBUG:      sync/:52:_sync(): force_dest_file = False
DEBUG:        sync/file/remote/:31:getPath(): Joining: '/' with 'foo'
DEBUG:        sync/file/remote/:34:getPath():  * got: 'drive://foo'
DEBUG:        sync/file/remote/:31:getPath(): Joining: '/' with 'foo'
DEBUG:        sync/file/remote/:34:getPath():  * got: 'drive://foo'
DEBUG:       sync/file/remote/:56:getInfo(): Fetching remote file metadata: 'drive://foo'
DEBUG:        drive/:462:stat(): Checking pcache for path: 'drive://foo'
DEBUG:        drive/:488:stat(): Got 2 paths from pathlist('drive://foo')
DEBUG:        drive/:489:stat(): Got paths: ['drive://', 'foo']
DEBUG:        drive/:497:stat(): Searching for 'foo' in path 'drive://'
DEBUG:        drive/:503:stat(): Checking pcache for path: 'drive://foo'
DEBUG:        drive/:506:stat():  * nothing found
DEBUG:         drive/:758:_query(): Checking gcache for parentId: root
DEBUG:           drive/:308:_getCredentialStorage(): Loading storage
DEBUG:             drive/:284:_getConfigDir(): Config dir = /home/kevin/.gsync
DEBUG:            drive/:300:_getConfigFile(): Environment: GSYNC_CREDENTIALS=/home/kevin/.gsync/credentials
DEBUG:          drive/:233:service(): Authenticating
DEBUG:           drive/:284:_getConfigDir(): Config dir = /home/kevin/.gsync
DEBUG:          drive/:242:service(): Loading Google Drive service from config
DEBUG:          drive/:246:service(): Downloading API service
DEBUG:          drive/:263:service(): Building Google Drive service from document
DEBUG:         drive/:788:_query(): Executing query: {'q': '"root" in parents and trashed = false'}
DEBUG:         drive/:792:_query(): Query returned 4 files
DEBUG:         drive/:799:_query(): Updating google cache: root (16 items)
DEBUG:        drive/:509:stat(): Got 16 entities back
DEBUG:         drive/:445:_findEntity(): Iterating 16 entities to find 'foo'
DEBUG:         drive/:450:_findEntity(): comparing u'foo' to u'bin'
DEBUG:         drive/:450:_findEntity(): comparing u'foo' to u'Music'
DEBUG:         drive/:450:_findEntity(): comparing u'foo' to u'Public'
DEBUG:         drive/:450:_findEntity(): comparing u'foo' to u'Pictures'
DEBUG:       sync/file/remote/:63:getInfo(): File not found: 'drive://foo'
DEBUG:      sync/:65:_sync(): Defaulting destination directory: 'drive://foo'
DEBUG:      sync/:71:_sync(): File not found: 'drive://foo'
cd+++++++++ foo/
DEBUG:        sync/file/:280:_normaliseSource(): type(src) = <class 'libgsync.sync.file.SyncFileInfo'>
DEBUG:         sync/file/factory.py:10:create(): SyncFileFactory.create('./foo')
DEBUG:         sync/file/factory.py:25:create(): Creating SyncFileLocal('foo')
DEBUG:         sync/file/remote/:73:_createDir(): Creating remote directory: 'drive://foo'
DEBUG:          drive/:537:mkdir(): path = 'drive://foo'
DEBUG:          drive/:544:mkdir(): spath = '/foo'
DEBUG:          drive/:545:mkdir(): normpath = 'drive://foo'
DEBUG:          drive/:550:mkdir(): dirname = 'drive:', basename = 'foo'
DEBUG:          drive/:569:mkdir(): Creating directory: 'drive://foo'
DEBUG:           drive/:728:_clearCache(): Clearing path cache entries for ''drive://foo''...
DEBUG:            drive/:462:stat(): Checking pcache for path: 'drive://foo'
DEBUG:            drive/:488:stat(): Got 2 paths from pathlist('drive://foo')
DEBUG:            drive/:489:stat(): Got paths: ['drive://', 'foo']
DEBUG:            drive/:497:stat(): Searching for 'foo' in path 'drive://'
DEBUG:            drive/:503:stat(): Checking pcache for path: 'drive://foo'
DEBUG:            drive/:506:stat():  * nothing found
DEBUG:             drive/:758:_query(): Checking gcache for parentId: root
DEBUG:            drive/:509:stat(): Got 16 entities back
DEBUG:             drive/:445:_findEntity(): Iterating 16 entities to find 'foo'
DEBUG:         drive/:450:_findEntity(): comparing u'foo' to u'bin'
DEBUG:         drive/:450:_findEntity(): comparing u'foo' to u'Music'
DEBUG:         drive/:450:_findEntity(): comparing u'foo' to u'Public'
DEBUG:         drive/:450:_findEntity(): comparing u'foo' to u'Pictures'
DEBUG:          sync/file/local/:30:getInfo(): Fetching local file metadata: 'foo'
DEBUG:          sync/file/local/:59:getInfo(): Local mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 11, 33, 12, 969249, tzinfo=tzutc()))
DEBUG:          sync/file/local/:64:getInfo(): Local mtime: 2013-11-30T11:33:12.969249+00:00
DEBUG:         sync/file/:242:__updateStats(): Updating: 'drive://foo'
DEBUG:         sync/file/:270:__updateStats():  * Updating with mtime: 1385811921.39
DEBUG:         sync/file/:271:__updateStats():  * Updating with atime: 1385811201.20
DEBUG:          sync/file/remote/:122:_updateStats(): Updating remote file stats: 'drive://foo'
DEBUG:            sync/file/remote/:31:getPath(): Joining: '/' with '/foo'
DEBUG:            sync/file/remote/:34:getPath():  * got: 'drive://foo'
DEBUG:           sync/file/remote/:56:getInfo(): Fetching remote file metadata: 'drive://foo'
DEBUG:            drive/:462:stat(): Checking pcache for path: 'drive://foo'
DEBUG:            drive/:466:stat(): Found path in path cache: 'drive://foo'
DEBUG:           sync/file/remote/:68:getInfo(): Remote mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 11, 45, 20, 997000, tzinfo=tzutc()))
DEBUG:            drive/:462:stat(): Checking pcache for path: 'drive://foo'
DEBUG:            drive/:466:stat(): Found path in path cache: 'drive://foo'
DEBUG:           drive/:681:update(): Updating: 'drive://foo'
DEBUG:           drive/:688:update():  * with: 'description' = 'eJxLzi/mis9NzE6NLy5JLIkvSi0uzSnhKjDk0tDwNODCRG6GxhamFoaGRgaGeoaW5kYWlkZQmRKNlAIjrmD14pL45JLM3FR1rgJjkHgxWKgoJbUMKGLC5Qfh50KVmCKUJOXkJ2cXA8XMYIoSoYrMsVoL05VdnFkFUmUB1FYSVGDJpQcAUdQ7Aw=='
DEBUG:           drive/:688:update():  * with: 'modifiedDate' = '2013-11-30T11:45:21.385923'
DEBUG:           drive/:691:update(): mdeia_body type = <type 'NoneType'>
DEBUG:    crawler.py:123:_walk(): Synchronising file: 'foo/bar'
DEBUG:      sync/:34:_sync(): Synchronising: 'foo/bar'
DEBUG:       sync/file/:338:relativeTo(): Creating relative path from '^./+' and 'foo/bar'
DEBUG:      sync/:37:_sync(): Destination: SyncFileRemote('drive://')
DEBUG:      sync/:38:_sync(): Relative: 'foo/bar'
DEBUG:        sync/file/:193:getPath(): Joining: '.' with 'foo/bar'
DEBUG:       sync/file/local/:30:getInfo(): Fetching local file metadata: './foo/bar'
DEBUG:       sync/file/local/:59:getInfo(): Local mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 11, 33, 12, 969249, tzinfo=tzutc()))
DEBUG:       sync/file/local/:64:getInfo(): Local mtime: 2013-11-30T11:33:12.969249+00:00
DEBUG:      sync/:52:_sync(): force_dest_file = False
DEBUG:        sync/file/remote/:31:getPath(): Joining: '/' with 'foo/bar'
DEBUG:        sync/file/remote/:34:getPath():  * got: 'drive://foo/bar'
DEBUG:        sync/file/remote/:31:getPath(): Joining: '/' with 'foo/bar'
DEBUG:        sync/file/remote/:34:getPath():  * got: 'drive://foo/bar'
DEBUG:       sync/file/remote/:56:getInfo(): Fetching remote file metadata: 'drive://foo/bar'
DEBUG:        drive/:462:stat(): Checking pcache for path: 'drive://foo/bar'
DEBUG:        drive/:488:stat(): Got 3 paths from pathlist('drive://foo/bar')
DEBUG:        drive/:489:stat(): Got paths: ['drive://', 'foo', 'bar']
DEBUG:        drive/:497:stat(): Searching for 'foo' in path 'drive://'
DEBUG:        drive/:503:stat(): Checking pcache for path: 'drive://foo'
DEBUG:        drive/:497:stat(): Searching for 'bar' in path 'drive://foo'
DEBUG:        drive/:503:stat(): Checking pcache for path: 'drive://foo/bar'
DEBUG:        drive/:506:stat():  * nothing found
DEBUG:         drive/:758:_query(): Checking gcache for parentId: 0B_9WraFp7iOpRFE5T0lBcWpnTVk
DEBUG:         drive/:788:_query(): Executing query: {'q': '"0B_9WraFp7iOpRFE5T0lBcWpnTVk" in parents and trashed = false'}
DEBUG: IOError(36, 'File name too long'):   File "/usr/local/lib/python2.7/dist-packages/libgsync/crawler.py", line 150, in run
    self._walk(srcpath, self._walkCallback, self._dev)
  File "/usr/local/lib/python2.7/dist-packages/libgsync/crawler.py", line 124, in _walk
    self._sync(f)
  File "/usr/local/lib/python2.7/dist-packages/libgsync/sync/__init__.py", line 31, in __call__
    self._sync(path)
  File "/usr/local/lib/python2.7/dist-packages/libgsync/sync/__init__.py", line 64, in _sync
    dstFile = self.dst.getInfo(relPath)
  File "/usr/local/lib/python2.7/dist-packages/libgsync/sync/file/remote/__init__.py", line 61, in getInfo
    info = drive.stat(path)
  File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 507, in stat
    ents = self._query(parentId = parentId)
  File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 790, in _query
    files = service.files().list(**param).execute()
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 716, in execute
    body=self.body, headers=self.headers)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 490, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1375, in _request
    _updateCache(headers, response, content, self.cache, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 448, in _updateCache
    cache.set(cachekey, text)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 715, in set
    f = file(cacheFullPath, "wb")

Error: IOError(36, 'File name too long')
DEBUG:  /usr/local/bin/gsync:66:<module>(): Crawlers finished, exiting
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
lionandoil commented 10 years ago

Just figured out that it works if I omit the root-slash in the Drive path, i.e. if I do any of

gsync -r foo drive://
gsync -r foo/ drive://foo
cd foo & gsync -r . drive://foo

Which one is the intended way to specify drive paths, with our without the initial slash? The only example usage in --help is with the preceding slash, it's strange that it accepts it first but then fails once it goes one directory deeper...

The first time (and only the first time) I do the upload/file creation it still spits out the error message mentioned in #30 - it doesn't just print it out in --debug though, but it goes to stderr as well, despite the synchronisation/upload succeeding. Here's the debug output from where it starts to differ from the one above, i.e. after it successfully created foo3/ remotely:

DEBUG:    crawler.py:123:_walk(): Synchronising file: 'foo3/bar'
DEBUG:      sync/:34:_sync(): Synchronising: 'foo3/bar'
DEBUG:       sync/file/:338:relativeTo(): Creating relative path from '^./+' and 'foo3/bar'
DEBUG:      sync/:37:_sync(): Destination: SyncFileRemote('drive://')
DEBUG:      sync/:38:_sync(): Relative: 'foo3/bar'
DEBUG:        sync/file/:193:getPath(): Joining: '.' with 'foo3/bar'
DEBUG:       sync/file/local/:30:getInfo(): Fetching local file metadata: './foo3/bar'
DEBUG:       sync/file/local/:59:getInfo(): Local mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 14, 21, 12, 160559, tzinfo=tzutc()))
DEBUG:       sync/file/local/:64:getInfo(): Local mtime: 2013-11-30T14:21:12.160559+00:00
DEBUG:      sync/:52:_sync(): force_dest_file = False
DEBUG:        sync/file/remote/:31:getPath(): Joining: '/' with 'foo3/bar'
DEBUG:        sync/file/remote/:34:getPath():  * got: 'drive://foo3/bar'
DEBUG:        sync/file/remote/:31:getPath(): Joining: '/' with 'foo3/bar'
DEBUG:        sync/file/remote/:34:getPath():  * got: 'drive://foo3/bar'
DEBUG:       sync/file/remote/:56:getInfo(): Fetching remote file metadata: 'drive://foo3/bar'
DEBUG:        drive/:462:stat(): Checking pcache for path: 'drive://foo3/bar'
DEBUG:        drive/:488:stat(): Got 3 paths from pathlist('drive://foo3/bar')
DEBUG:        drive/:489:stat(): Got paths: ['drive://', 'foo3', 'bar']
DEBUG:        drive/:497:stat(): Searching for 'foo3' in path 'drive://'
DEBUG:        drive/:503:stat(): Checking pcache for path: 'drive://foo3'
DEBUG:        drive/:497:stat(): Searching for 'bar' in path 'drive://foo3'
DEBUG:        drive/:503:stat(): Checking pcache for path: 'drive://foo3/bar'
DEBUG:        drive/:506:stat():  * nothing found
DEBUG:         drive/:758:_query(): Checking gcache for parentId: 0B_9WraFp7iOpRmg5Slg4elNrRlk
DEBUG:         drive/:788:_query(): Executing query: {'q': '"0B_9WraFp7iOpRmg5Slg4elNrRlk" in parents and trashed = false'}
DEBUG:         drive/:792:_query(): Query returned 4 files
DEBUG:         drive/:799:_query(): Updating google cache: 0B_9WraFp7iOpRmg5Slg4elNrRlk (0 items)
DEBUG:        drive/:509:stat(): Got 0 entities back
DEBUG:       sync/file/remote/:63:getInfo(): File not found: 'drive://foo3/bar'
DEBUG:      sync/:65:_sync(): Defaulting destination directory: 'drive://foo3/bar'
DEBUG:      sync/:71:_sync(): File not found: 'drive://foo3/bar'
DEBUG:       sync/file/:347:islocal(): self.__class__.__name__ == SyncFileRemote
foo3/bar
DEBUG:        sync/file/:280:_normaliseSource(): type(src) = <class 'libgsync.sync.file.SyncFileInfo'>
DEBUG:         sync/file/factory.py:10:create(): SyncFileFactory.create('./foo3/bar')
DEBUG:         sync/file/factory.py:25:create(): Creating SyncFileLocal('foo3/bar')
DEBUG:         sync/file/remote/:80:_createFile(): Creating remote file: 'drive://foo3/bar'
DEBUG:          sync/file/local/:30:getInfo(): Fetching local file metadata: 'foo3/bar'
DEBUG:          sync/file/local/:59:getInfo(): Local mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 14, 21, 12, 160559, tzinfo=tzutc()))
DEBUG:          sync/file/local/:64:getInfo(): Local mtime: 2013-11-30T14:21:12.160559+00:00
DEBUG:           drive/:462:stat(): Checking pcache for path: 'drive://foo3'
DEBUG:           drive/:466:stat(): Found path in path cache: 'drive://foo3'
DEBUG:           drive/:462:stat(): Checking pcache for path: 'drive://foo3/bar'
DEBUG:           drive/:488:stat(): Got 3 paths from pathlist('drive://foo3/bar')
DEBUG:           drive/:489:stat(): Got paths: ['drive://', 'foo3', 'bar']
DEBUG:           drive/:497:stat(): Searching for 'foo3' in path 'drive://'
DEBUG:           drive/:503:stat(): Checking pcache for path: 'drive://foo3'
DEBUG:           drive/:497:stat(): Searching for 'bar' in path 'drive://foo3'
DEBUG:           drive/:503:stat(): Checking pcache for path: 'drive://foo3/bar'
DEBUG:           drive/:506:stat():  * nothing found
DEBUG:            drive/:758:_query(): Checking gcache for parentId: 0B_9WraFp7iOpRmg5Slg4elNrRlk
DEBUG:           drive/:509:stat(): Got 0 entities back
DEBUG:           drive/:728:_clearCache(): Clearing path cache entries for ''drive://foo3/bar''...
DEBUG:            drive/:462:stat(): Checking pcache for path: 'drive://foo3/bar'
DEBUG:            drive/:488:stat(): Got 3 paths from pathlist('drive://foo3/bar')
DEBUG:            drive/:489:stat(): Got paths: ['drive://', 'foo3', 'bar']
DEBUG:            drive/:497:stat(): Searching for 'foo3' in path 'drive://'
DEBUG:            drive/:503:stat(): Checking pcache for path: 'drive://foo3'
DEBUG:            drive/:497:stat(): Searching for 'bar' in path 'drive://foo3'
DEBUG:            drive/:503:stat(): Checking pcache for path: 'drive://foo3/bar'
DEBUG:            drive/:506:stat():  * nothing found
DEBUG:             drive/:758:_query(): Checking gcache for parentId: 0B_9WraFp7iOpRmg5Slg4elNrRlk
DEBUG:            drive/:509:stat(): Got 0 entities back
DEBUG:         sync/file/remote/:91:_updateFile(): Updating remote file: 'drive://foo3/bar'
DEBUG:          sync/file/local/:30:getInfo(): Fetching local file metadata: 'foo3/bar'
DEBUG:          sync/file/local/:59:getInfo(): Local mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 14, 21, 12, 160559, tzinfo=tzutc()))
DEBUG:          sync/file/local/:64:getInfo(): Local mtime: 2013-11-30T14:21:12.160559+00:00
DEBUG:           sync/file/local/:30:getInfo(): Fetching local file metadata: 'foo3/bar'
DEBUG:           sync/file/local/:59:getInfo(): Local mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 14, 21, 12, 160559, tzinfo=tzutc()))
DEBUG:           sync/file/local/:64:getInfo(): Local mtime: 2013-11-30T14:21:12.160559+00:00
DEBUG:           drive/:462:stat(): Checking pcache for path: 'drive://foo3/bar'
DEBUG:           drive/:466:stat(): Found path in path cache: 'drive://foo3/bar'
DEBUG:          drive/:681:update(): Updating: 'drive://foo3/bar'
DEBUG:          drive/:688:update():  * with: 'mimeType' = 'none/unknown-mimetype'
DEBUG:          drive/:688:update():  * with: 'fileSize' = 0
DEBUG:          drive/:688:update():  * with: 'modifiedDate' = SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 14, 21, 12, 160559, tzinfo=tzutc()))
DEBUG:          drive/:688:update():  * with: 'statInfo' = posix.stat_result(st_mode=33188, st_ino=7865630, st_dev=64515L, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1385821290, st_mtime=1385821272, st_ctime=1385821272)
DEBUG:          drive/:688:update():  * with: 'title' = 'bar'
DEBUG:          drive/:688:update():  * with: 'path' = 'foo3/bar'
DEBUG:          drive/:688:update():  * with: 'md5Checksum' = None
DEBUG:          drive/:688:update():  * with: 'description' = 'eJx9j8sKg0AMRff5EV3JPMw8fqAguGo/YLA6C1FxMGMX/fqOrcW6KYRwuZwTSDsTuKkZvKPYRLd4WscIgUOeV1JyY6DSRqGSDGpVIscaKp6GMfbdW5AGjeDC/mQtTjnmXRBwyyi6NvaTzyBIuBxAwRVDNBrozSydfySk3M5/mmm38I91H+d2oASpw2t2Tx+eZYVAcfIG6p8blf4tmVVA8RosFC/Pt0ml'
DEBUG:          drive/:691:update(): mdeia_body type = <class 'apiclient.http.MediaFileUpload'>
DEBUG:          drive/:708:update():  * uploading next chunk...
DEBUG:          drive/:713:update(): Exception: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v2/files/0B_9WraFp7iOpV3Myb3BIR1BPY1E?uploadType=resumable&alt=json&newRevision=true&setModifiedDate=false returned "Invalid Upload Request">
DEBUG: 'Exception':   File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 711, in update
    status, res = req.next_chunk()
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 874, in next_chunk
    return self._process_response(resp, content)
  File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 901, in _process_response
    raise HttpError(resp, content, uri=self.uri)

DEBUG:         sync/file/remote/:116:_updateFile(): Update failed
DEBUG:          sync/file/local/:30:getInfo(): Fetching local file metadata: 'foo3/bar'
DEBUG:          sync/file/local/:59:getInfo(): Local mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 14, 21, 12, 160559, tzinfo=tzutc()))
DEBUG:          sync/file/local/:64:getInfo(): Local mtime: 2013-11-30T14:21:12.160559+00:00
DEBUG:         sync/file/:242:__updateStats(): Updating: 'drive://foo3/bar'
DEBUG:         sync/file/:270:__updateStats():  * Updating with mtime: 1385821420.38
DEBUG:         sync/file/:271:__updateStats():  * Updating with atime: 1385821290.25
DEBUG:          sync/file/remote/:122:_updateStats(): Updating remote file stats: 'drive://foo3/bar'
DEBUG:            sync/file/remote/:31:getPath(): Joining: '/' with '/foo3/bar'
DEBUG:            sync/file/remote/:34:getPath():  * got: 'drive://foo3/bar'
DEBUG:           sync/file/remote/:56:getInfo(): Fetching remote file metadata: 'drive://foo3/bar'
DEBUG:            drive/:462:stat(): Checking pcache for path: 'drive://foo3/bar'
DEBUG:            drive/:466:stat(): Found path in path cache: 'drive://foo3/bar'
DEBUG:           sync/file/remote/:68:getInfo(): Remote mtime: SyncFileInfoDatetime(datetime.datetime(2013, 11, 30, 14, 21, 12, 160000, tzinfo=tzutc()))
DEBUG:            drive/:462:stat(): Checking pcache for path: 'drive://foo3/bar'
DEBUG:            drive/:466:stat(): Found path in path cache: 'drive://foo3/bar'
DEBUG:           drive/:681:update(): Updating: 'drive://foo3/bar'
DEBUG:           drive/:688:update():  * with: 'description' = 'eJxtjk0Kg0AMRve5iK5kfsxMvEBBkC7aAwxWZyEqDk7aRU/faasLQQghyXsfpFsiuLkdvYvcslt9fE4MQUKe11pLIqgtGTRaQGNKlNhALVMJIfYu4CI1ISmpKlEoVIhkE9xuVh1mzvug4J5Fdh0Ps88g6IMQf2zt/SuhEq7/fd5cPHEf09KNMUGz2+1m29PX9tQYh/fXohTjW6ig+ABwSkbK'
DEBUG:           drive/:688:update():  * with: 'modifiedDate' = '2013-11-30T14:23:40.383411'
DEBUG:           drive/:691:update(): mdeia_body type = <type 'NoneType'>
sent 0 bytes  received 0 bytes  0.00 bytes/sec
DEBUG:  /usr/local/bin/gsync:66:<module>(): Crawlers finished, exiting
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
iwonbigbro commented 10 years ago

This does look like a new bug. I'll look into this asap.

iwonbigbro commented 10 years ago

The path munging is supposed to strip off the drive:// part leaving just the absolute path under Google Drive root. However, if something is failing with this, it might be generating an invalid path. I'll take a look through the debug output at my earliest opportunity.

iwonbigbro commented 10 years ago

Wow, this is interesting. The file being opened (with the filename that is too long) is the httplib local cache. I have no idea at this stage, why omitting the extra slash has anything to do with the cache name that is generated, since the filenames have nothing to do with the URI in google drive. Files are all accessed via a UUID.

So the API query made in the first (failing) run:

{'q': '"0B_9WraFp7iOpRFE5T0lBcWpnTVk" in parents and trashed = false'}

The query in the second (working) run:

{'q': '"0B_9WraFp7iOpRmg5Slg4elNrRlk" in parents and trashed = false'}

A subtle difference in the ID being requested, which is odd. I will see if there is a known issue with httplib...

iwonbigbro commented 10 years ago

https://code.google.com/p/httplib2/issues/detail?id=92

Looks like I could do something to mitigate the issue.

PaoloneM commented 10 years ago

Hi,

same problem, with or without extra slash... Running on Ubuntu server 12.04 LTS, Python 2.7.3, Gsync 0.1.14 installed git cloning. Any suggested workaround?

DEBUG: IOError(36, 'File name too long'): [Errno 36] File name too long: '/home/paolo/.gsync/http_cache/www.googleapis.com,drive,v2,files,q=%22root%22+in+parents+and+trashed+%3D+false&pageToken=EAIaowELEgBSmgEKkAEKafjzZuv_____wxLMErNCyUAA_wH__kNvc21vLnVzZXIoMDAwMDAwNWYwODVlNTdiYVUpLmRpcl9lbnRyeSg0MDgxNj,aa07a32f36e29b0562bfada40b080270'
DEBUG: IOError(36, 'File name too long'):   File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/EGG-INFO/scripts/gsync", line 82, in main
    Crawler(src, dest).run()
  File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/crawler.py", line 76, in __init__
    info = self._drive.stat(self._dst)
  File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/drive/__init__.py", line 665, in stat
    ents = self._query(parent_id=parent_id)
  File "/usr/local/lib/python2.7/dist-packages/retrying-1.2.3-py2.7.egg/retrying.py", line 92, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/retrying-1.2.3-py2.7.egg/retrying.py", line 253, in call
    raise attempt.get()
  File "/usr/local/lib/python2.7/dist-packages/retrying-1.2.3-py2.7.egg/retrying.py", line 285, in get
    reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/local/lib/python2.7/dist-packages/retrying-1.2.3-py2.7.egg/retrying.py", line 241, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/drive/__init__.py", line 939, in _query
    files = service.files().list(**param).execute()
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 716, in execute
    body=self.body, headers=self.headers)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 490, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1593, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1396, in _request
    _updateCache(headers, response, content, self.cache, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 448, in _updateCache
    cache.set(cachekey, text)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 715, in set
    f = file(cacheFullPath, "wb")

gsync: [Errno 36] File name too long: '/home/paolo/.gsync/http_cache/www.googleapis.com,drive,v2,files,q=%22root%22+in+parents+and+trashed+%3D+false&pageToken=EAIaowELEgBSmgEKkAEKafjzZuv_____wxLMErNCyUAA_wH__kNvc21vLnVzZXIoMDAwMDAwNWYwODVlNTdiYVUpLmRpcl9lbnRyeSg0MDgxNj,aa07a32f36e29b0562bfada40b080270'
gsync error: IOError at drive(939) [client=0.1.14]

Can I help you in debugging / fixing issue?

Regards.

Paolo

42kohd commented 9 years ago

same issue here

DEBUG:       drive/:284:_getConfigDir(): Config dir = /home/jeremy/.gsync
DEBUG:      drive/:242:service(): Loading Google Drive service from config
DEBUG:      drive/:246:service(): Downloading API service
DEBUG:      drive/:263:service(): Building Google Drive service from document
DEBUG:     drive/:788:_query(): Executing query: {'q': '"root" in parents and trashed = false'}
DEBUG:     drive/:792:_query(): Query returned 4 files
DEBUG:     drive/:799:_query(): Updating google cache: root (35 items)
DEBUG:    drive/:509:stat(): Got 35 entities back
DEBUG:     drive/:445:_findEntity(): Iterating 35 entities to find 'backups'
DEBUG:     drive/:450:_findEntity(): comparing u'backups' to u'backups'
DEBUG:     drive/:452:_findEntity(): Found u'backups'
DEBUG:    drive/:497:stat(): Searching for 'rawphotos' in path 'drive://backups'
DEBUG:    drive/:503:stat(): Checking pcache for path: 'drive://backups/rawphotos'
DEBUG:    drive/:506:stat():  * nothing found
DEBUG:     drive/:758:_query(): Checking gcache for parentId: 0B55M-kBru5L2WnJNaVdNbGtjajQ
DEBUG:     drive/:788:_query(): Executing query: {'q': '"0B55M-kBru5L2WnJNaVdNbGtjajQ" in parents and trashed = false'}
DEBUG: IOError(36, 'File name too long'):   File "/usr/local/bin/gsync", line 55, in <module>
    crawler = Crawler(p, dest)
  File "/usr/local/lib/python2.7/dist-packages/libgsync/crawler.py", line 57, in __init__
    info = self._drive.stat(self._dst)
  File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 507, in stat
    ents = self._query(parentId = parentId)
  File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 790, in _query
    files = service.files().list(**param).execute()
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 722, in execute
    body=self.body, headers=self.headers)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 551, in new_request
    redirections, connection_type)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1569, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1374, in _request
    _updateCache(headers, response, content, self.cache, cachekey)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 447, in _updateCache
    cache.set(cachekey, text)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 714, in set
    f = file(cacheFullPath, "wb")

Error: [Errno 36] File name too long: '/home/jeremy/.gsync/http_cache/www.googleapis.com,drive,v2,files,q=%220B55M-kBru5L2WnJNaVdNbGtjajQ%22+in+parents+and+trashed+%3D+false&alt=json,1f03b05e5315d003d9962037dad51472'
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
42kohd commented 9 years ago

paths don't matter I can't get anything to happen.

42kohd commented 9 years ago

I believe I fixed mine, seems to have been due to an outdated httplib2 which was not upgraded due to a permissions problem.