search

havardgulldahl / jottalib

A library to access files stored at jottacloud.com.
GNU General Public License v3.0
83 stars 19 forks source link

jotta-scanner fail to upload file #138

Open EigilSletbak opened 7 years ago

EigilSletbak commented 7 years ago

PC folder content: eisl@EISL01 ~ $ ls -l /home/eisl/Documents total 8 drwx------ 2 eisl eisl 4096 Sep 22 11:32 NoMachine -rw-r--r-- 1 eisl eisl 34 Jan 4 13:52 Test.txt

Run jotta-scanner eisl@EISL01 ~ $ jotta-scanner /home/eisl/Documents /Jotta/Archive/EISL01 Entering dir: /home/eisl/Documents uploading 1 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 1/1 - 00:00:01 Network upload speed 33.543B/sec Entering dir: /home/eisl/Documents/NoMachine Finished syncing 1 files to JottaCloud, no errors. yay!

Web Client show empty Docments folder image

Folder checked as mounted using jotta-fuse, showing 0 size file eisl@EISL01 ~ $ ls -l /home/eisl/jottafs/Jotta/Archive/EISL01/Documents total 0 -rw-r--r-- 0 eisl eisl 0 Jan 5 14:26 Test.txt

Removes the Documents folder using the Web Interface. eisl@EISL01 ~ $ ls -l /home/eisl/jottafs/Jotta/Archive/EISL01 total 0

Running with debug on eisl@EISL01 ~ $ jotta-scanner -l debug /home/eisl/Documents /Jotta/Archive/EISL01 DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com', extra_headers=None, params=None DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.jottacloud.com send: 'GET /jfs/eigil.sletbak@gmail.com HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Thu, 05 Jan 2017 13:28:55 GMT header: X-ID: 400889492780 header: Content-Type: text/xml header: Transfer-Encoding: chunked header: Server: Jetty(8.1.21.v20160908) header: Strict-Transport-Security: max-age= 31536000 DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com HTTP/1.1" 200 None INFO:root:args: topdir u'/home/eisl/Documents', jottapath u'/Jotta/Archive/EISL01' DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem(u'/home/eisl/Documents') DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem('/home/eisl/Documents') DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('/home/eisl/Documents') DEBUG:jottalib.jottacloud:compare walk: u'/home/eisl/Documents' -> 1 files DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('Test.txt') DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('NoMachine') DEBUG:jottalib.jottacloud:get_jottapath u'/home/eisl/Documents' u'/home/eisl/Documents' u'/Jotta/Archive/EISL01' DEBUG:jottalib.jottacloud:_decode_filename_to_unicode(u'/Jotta/Archive/EISL01/Documents') DEBUG:jottalib.jottacloud:compare jottapath: u'/Jotta/Archive/EISL01/Documents' DEBUG:jottalib.jottacloud:filelist u'/Jotta/Archive/EISL01/Documents' DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents', extra_headers=None, params=None send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Thu, 05 Jan 2017 13:28:56 GMT header: Accept-Ranges: bytes header: X-ID: 120555229075 header: Content-Type: text/xml header: Transfer-Encoding: chunked header: Server: Jetty(8.1.21.v20160908) header: Strict-Transport-Security: max-age= 31536000 DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1" 200 None DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents' INFO:jottalib.JFS:syncing u'/Jotta/Archive/EISL01/Documents' DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents' DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents', extra_headers=None, params=None send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Thu, 05 Jan 2017 13:28:56 GMT header: Accept-Ranges: bytes header: X-ID: 921222432945 header: Content-Type: text/xml header: Transfer-Encoding: chunked header: Server: Jetty(8.1.21.v20160908) header: Strict-Transport-Security: max-age= 31536000 DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1" 200 None DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents' DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents' DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents' DEBUG:root:folderlist u'/Jotta/Archive/EISL01/Documents' DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents', extra_headers=None, params=None send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Thu, 05 Jan 2017 13:28:56 GMT header: Accept-Ranges: bytes header: X-ID: 460642854897 header: Content-Type: text/xml header: Transfer-Encoding: chunked header: Server: Jetty(8.1.21.v20160908) header: Strict-Transport-Security: max-age= 31536000 DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1" 200 None DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents' INFO:jottalib.JFS:syncing u'/Jotta/Archive/EISL01/Documents' DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents' DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents', extra_headers=None, params=None send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Thu, 05 Jan 2017 13:28:56 GMT header: Accept-Ranges: bytes header: X-ID: 694074858768 header: Content-Type: text/xml header: Transfer-Encoding: chunked header: Server: Jetty(8.1.21.v20160908) header: Strict-Transport-Security: max-age= 31536000 DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1" 200 None DEBUG:jottalib.jottacloud:--cloudfiles: set([]) DEBUG:jottalib.jottacloud:--localfiles: set(['Test.txt']) DEBUG:jottalib.jottacloud:--cloudfolders: set([]) DEBUG:jottalib.jottacloud:Create SyncFile from 'Test.txt' DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem('Test.txt') DEBUG:jottalib.jottacloud:Got encoded filename 'Test.txt', joining with dirpath '/home/eisl/Documents' DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem('Test.txt') DEBUG:jottalib.jottacloud:_decode_filename_to_unicode(u'/Jotta/Archive/EISL01/Documents') DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('Test.txt') Entering dir: /home/eisl/Documents DEBUG:jottalib.scanner:uploading new file: SyncFile(localpath='/home/eisl/Documents/Test.txt', jottapath=u'/Jotta/Archive/EISL01/Documents/Test.txt') DEBUG:jottalib.scanner:running <function new at 0x7f7d55467230> with args ('/home/eisl/Documents/Test.txt', u'/Jotta/Archive/EISL01/Documents/Test.txt', <jottalib.JFS.JFS object at 0x7f7d55486d50>) DEBUG:jottalib.JFS:posting content (len 34, hash a8b3b313e7685d4c400c0816f3b4c8f4) to url u'/Jotta/Archive/EISL01/Documents/Test.txt' DEBUG:jottalib.JFS:posting content (len ?) to url https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/Test.txt send: 'POST /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/Test.txt?cphash=a8b3b313e7685d4c400c0816f3b4c8f4 HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\njx_lisence: \r\nJMd5: a8b3b313e7685d4c400c0816f3b4c8f4\r\nJModified: 2017-01-04T13:52:49.311838\r\nJSize: 34\r\nX-Jfs-DeviceName: Jotta\r\nJCreated: 2017-01-04T13:52:49.311838\r\ncontent-type: multipart/form-data; boundary=932bcbb48e2d421994d6b8d4ae39c9f9\r\njx_csid: \r\nContent-Length: 602\r\nAuthorization: Basic XXXX\r\n\r\n' send: <MultipartEncoder: {'created': ('', '2017-01-04T13:52:49.311838'), 'modified': ('', '2017-01-04T13:52:49.311838'), 'file': (u'Test.txt', <open file '/home/eisl/Documents/Test.txt', mode 'r' at 0x7f7d5546b0c0>, 'application/octet-stream'), 'md5': ('', 'a8b3b313e7685d4c400c0816f3b4c8f4')}> sendIng a read()able reply: 'HTTP/1.1 201 Created\r\n' header: Date: Thu, 05 Jan 2017 13:28:56 GMT header: X-ID: 525790647848 header: Content-Type: text/xml header: Transfer-Encoding: chunked header: Server: Jetty(8.1.21.v20160908) header: Strict-Transport-Security: max-age= 31536000 DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "POST /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/Test.txt?cphash=a8b3b313e7685d4c400c0816f3b4c8f4 HTTP/1.1" 201 None uploading 1 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 1/1 - 00:00:00 Network upload speed 85.788B/sec DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem('/home/eisl/Documents/NoMachine') DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('/home/eisl/Documents/NoMachine') DEBUG:jottalib.jottacloud:compare walk: u'/home/eisl/Documents/NoMachine' -> 0 files DEBUG:jottalib.jottacloud:get_jottapath u'/home/eisl/Documents' u'/home/eisl/Documents/NoMachine' u'/Jotta/Archive/EISL01' DEBUG:jottalib.jottacloud:_decode_filename_to_unicode(u'/Jotta/Archive/EISL01/Documents/NoMachine') DEBUG:jottalib.jottacloud:compare jottapath: u'/Jotta/Archive/EISL01/Documents/NoMachine' DEBUG:jottalib.jottacloud:filelist u'/Jotta/Archive/EISL01/Documents/NoMachine' DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine', extra_headers=None, params=None send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n' reply: 'HTTP/1.1 404 Not Found\r\n' header: Date: Thu, 05 Jan 2017 13:28:57 GMT header: Accept-Ranges: bytes header: X-ID: 774401734012 header: Content-Type: text/xml header: Transfer-Encoding: chunked header: Server: Jetty(8.1.21.v20160908) header: Strict-Transport-Security: max-age= 31536000 DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine HTTP/1.1" 404 None DEBUG:root:folderlist u'/Jotta/Archive/EISL01/Documents/NoMachine' DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine', extra_headers=None, params=None send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n' reply: 'HTTP/1.1 404 Not Found\r\n' header: Date: Thu, 05 Jan 2017 13:28:57 GMT header: Accept-Ranges: bytes header: X-ID: 666925415241 header: Content-Type: text/xml header: Transfer-Encoding: chunked header: Server: Jetty(8.1.21.v20160908) header: Strict-Transport-Security: max-age= 31536000 DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine HTTP/1.1" 404 None DEBUG:jottalib.jottacloud:--cloudfiles: set([]) DEBUG:jottalib.jottacloud:--localfiles: set([]) DEBUG:jottalib.jottacloud:--cloudfolders: set([]) Entering dir: /home/eisl/Documents/NoMachine Finished syncing 1 files to JottaCloud, no errors. yay!

sencer commented 7 years ago

Same here.

Linux Sencer 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

sjurtf commented 7 years ago

What version of requests are you using?

It was recently descovered in #137 that newer versions of requests is breaking jottalib. The issue here looks like what was happening in #137.

EigilSletbak commented 7 years ago

Seems to be relevant, this is my requests info:

eisl@EISL01 ~ $ pip show requests Name: requests Version: 2.12.4 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: me@kennethreitz.com License: Apache 2.0 Location: /usr/local/lib/python2.7/dist-packages

sjurtf commented 7 years ago

If you downgrade to requests 2.12.0 it should work again.

EigilSletbak commented 7 years ago

Still fail with requests 2.12.0

eisl@EISL01 ~ $ pip show requests Name: requests Version: 2.12.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: me@kennethreitz.com License: Apache 2.0 Location: /usr/local/lib/python2.7/dist-packages Requires:

eisl@EISL01 ~ $ ls -l /home/eisl/Documents total 8 drwx------ 2 eisl eisl 4096 Sep 22 11:32 NoMachine -rw-r--r-- 1 eisl eisl 34 Jan 4 13:52 Test.txt

eisl@EISL01 ~ $ jotta-scanner /home/eisl/Documents /Jotta/Archive/EISL01 Entering dir: /home/eisl/Documents uploading 1 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 1/1 - 00:00:00 Network upload speed 119.602B/sec Entering dir: /home/eisl/Documents/NoMachine Finished syncing 1 files to JottaCloud, no errors. yay!

eisl@EISL01 ~ $ ls -l /home/eisl/jottafs/Jotta/Archive/EISL01/Documents total 0 -rw-r--r-- 0 eisl eisl 0 Jan 9 15:36 Test.txt

sencer commented 7 years ago

I also tried downgrading requests to 2.9.1, but it didn't work.

fholmer commented 7 years ago

I have same issue too. jotta-upload and jotta-fuse is also affected. I inspected the files status reported by jotta api and all uploaded files have incomplete status. I also compared the md5sum reported by jotta API and it matches md5sum of local files. So content is uploaded but file is still incomplete. It is not possible to list, download, delete og restore these files.

paaland commented 7 years ago

I too have this problem. Seems the files are there somewhere (lost in space), because running jotta-scanner multiple times it starts to compare existing files. So somehow jotta-scanner finds the previously uploaded files, but https://www.jottacloud.com does not. The folder structure gets created, but no files are visible on the web site.

$jotta-scanner --exclude .streams /mnt/ /QNAPNAS/Backup

Entering dir: /mnt/Documents
uploading 14 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 14/14 - 00:00:05
Network upload speed 650.720KiB/sec
comparing 31 existing files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 31/31 - 00:00:20
Entering dir: /mnt/Documents/Bilder
Entering dir: /mnt/Documents/Bilder/Polar campingvogn
uploading 2 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 2/2 - 00:00:00
Network upload speed 57.232KiB/sec
comparing 10 existing files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 10/10 - 00:00:05
comparing 2 existing files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 2/2 - 00:00:00
Entering dir: /mnt/Documents/Diverse
uploading 35 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 9/35 - 00:00:04
Finished syncing 82 files to JottaCloud, no errors. yay!
ghost commented 7 years ago

I have the same problem too: $ pip show requests Name: requests Version: 2.13.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: me@kennethreitz.com License: Apache 2.0 Location: /docker-data/_backup/pip/docker_backup/lib/python2.7/site-packages Requires:

sjurtf commented 7 years ago

We have found that at least these versions and combinations of libs are working together with jottalib. We have not had time to update one and one an check which is breaking it yet. 

certifi==2016.2.28
clint==0.5.1
humanize==0.5.1
lxml==3.7.2
six==1.10.0
chardet==2.3.0
python-dateutil==2.5.3
requests-toolbelt==0.6.2
jottalib==0.5.1
ghost commented 7 years ago

I tried using the versions listed above, but no luck. I have some other additional packages installed, not sure if that should make any difference: appdirs (1.4.0) args (0.1.0) backports.ssl-match-hostname (3.5.0.1) certifi (2016.2.28) cffi (1.9.1) chardet (2.3.0) clint (0.5.1) docker (2.0.2) docker-pycreds (0.2.1) humanize (0.5.1) ipaddress (1.0.18) jottalib (0.5.1) lxml (3.7.2) packaging (16.8) pip (9.0.1) pycparser (2.17) pyparsing (2.1.10) python-dateutil (2.5.3) python-gnupg (0.4.0) PyYAML (3.12) requests (2.13.0) requests-toolbelt (0.6.2) setuptools (34.1.1) six (1.10.0) websocket-client (0.40.0) wheel (0.29.0) xattr (0.9.1)

sjurtf commented 7 years ago

Try downgrading to requests 2.10.0. You can also try using my fork - it should not really matter with this issue tho. But Im using that one.

pip install git+https://github.com/sjurtf/jottalib.git

ghost commented 7 years ago

Ok. I have now tested a bit more. Downgrading worked. Yay! I only have requests==2.10.0 in requirements to make upload work again. Thanks everybody!