search

xybu / onedrive-d-old

Microsoft OneDrive client on Linux.
http://xybu.me/projects/onedrive-d/
GNU Lesser General Public License v3.0
820 stars 143 forks source link

Uncaught OneDriveAPIException exceptions #144

Open kind3r opened 9 years ago

kind3r commented 9 years ago

I'm trying to upload a large number of files (all my photos from G+ aprox 40G) to onedrive. The issue I encounter is that the upload task (not sure if that's the correct name) keeps stopping after a while. I used to restart onedrive-d and the upload went forward but after a while it stopped alltoghether. All the folders have synced (850) but not the files. In the log I only see added task "up" for files but no uploading block. One more thing worth mentioning is that the folder I'm trying to upload is symlinked inside ~/OneDrive.

How can I extract more debug information ? Is there a limit on the number of files onedrive-d can sync ?

xybu commented 9 years ago

For small files (size smaller than the BITS min file size parameter) the program uses HTTP PUT to upload the file and for large files the program uses MS BITS_POST which I found not as reliable as I assume.

HTTP PUT upload method does not have detailed log since it is assumed to succeed (unless IO Error). If you see a log saying "get" task of type "up" then some worker is uploading that file. If you see a line saying "add" log "up" then a new task is created and waits handling. You could try monitoring network traffic and see if things get sent out.

Another thing is that MS SkyDrive API (Live API) has a limit on how many requests can be made in a period of time. Current program in future branch could not handle it. It could crash workers one by one. 3rd party applications like onedrive-d suffer from more undocumented limits.

Next version of onedrive-d uses MS OneDrive API which was released about a month ago. While the new version is slowly in progress, the new OneDrive API is itself under testing. I can't guarantee an ETA for next version.

Sincerely, Xiangyu Bu

Sent from my Windows Phone

From: Emanuel Posescumailto:notifications@github.com Sent: ‎3/‎20/‎2015 3:16 AM To: xybu/onedrive-dmailto:onedrive-d@noreply.github.com Subject: [onedrive-d] Syncing large number of files (#144)

I'm trying to upload a large number of files (all my photos from G+ aprox 40G) to onedrive. The issue I encounter is that the upload task (not sure if that's the correct name) keeps stopping after a while. I used to restart onedrive-d and the upload went forward but after a while it stopped alltoghether. All the folders have synced (850) but not the files. In the log I only see added task "up" for files but no uploading block. One more thing worth mentioning is that the folder I'm trying to upload is symlinked inside ~/OneDrive.

How can I extract more debug information ? Is there a limit on the number of files onedrive-d can sync ?

Reply to this email directly or view it on GitHub: https://github.com/xybu/onedrive-d/issues/144

kind3r commented 9 years ago

That's the thing, I only have added task "up" on files and got task: sy on folders, so no got task: up on files. I will try to do a tcpdump and investigate the traffic.

xybu commented 9 years ago

Could you post some log? Preferably from MainThread start to the death of the program. Unnecessary stuff can be left out, but be sure to include anything about "exception". Don't use daemon mode btw.

Sincerely, Xiangyu Bu

Sent from my Windows Phone

From: Emanuel Posescumailto:notifications@github.com Sent: ‎3/‎20/‎2015 3:44 AM To: xybu/onedrive-dmailto:onedrive-d@noreply.github.com Cc: Xiangyu Bumailto:xybu92@live.com Subject: Re: [onedrive-d] Syncing large number of files (#144)

That's the thing, I only have added task "up" on files and got task: sy on folders, so no got task: up on files. I will try to do a tcpdump and investigate the traffic.

Reply to this email directly or view it on GitHub: https://github.com/xybu/onedrive-d/issues/144#issuecomment-83944172

kind3r commented 9 years ago

I think I'm getting somewhere. All 4 worker threads seem to get stuck on this exception (just different folder/resource names)

Exception in thread worker3: Traceback (most recent call last): File "/usr/lib/python3.3/threading.py", line 637, in _bootstrap_inner self.run() File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 443, in run self.make_remote_dir(task) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 338, in make_remote_dir new_entry = self.api.mkdir(name, task['remote_parent_id']) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 328, in mkdir return self.parse_response(r, OneDriveAPIException, requests.codes.created) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 115, in parse_response raise error(ret) onedrive_d.od_onedrive_api.OneDriveAPIException: The resource couldn't be created because a resource named '2013-09-14 -' already exists. (resource_already_exists)

xybu commented 9 years ago

So the remote dir did not exist when its parent was scanned, but was created before the "mkdir" task was performed. But at most one task can exist for a given local path.

May I know if any other onedrive client is running? It could be possible that another client created the dir before onedrive-d did.

But this is a problem. Should issue a fix to make the worker more robust.

kind3r commented 9 years ago

I don't know how this happened but it seems that I have duplicate folder names. I have no idea how this happened but there are 2 folders named '2013-09-14 -' with different content. Something must have gone wrong during the unzip of the archives I downloaded from Google with my backups. So it appears it's not really a onedrive-d issue (tho gracefully treating that exception would be nice to have).

kind3r commented 9 years ago

I have another exception that crashes the worker (I will rename the folder)

Exception in thread worker2: Traceback (most recent call last): File "/usr/lib/python3.3/threading.py", line 637, in _bootstrap_inner self.run() File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 443, in run self.make_remote_dir(task) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 338, in make_remote_dir new_entry = self.api.mkdir(name, task['remote_parent_id']) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 328, in mkdir return self.parse_response(r, OneDriveAPIException, requests.codes.created) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 115, in parse_response raise error(ret) onedrive_d.od_onedriveapi.OneDriveAPIException: The operation couldn't be completed because the requested folder name, 'Hangout Bogdan Postelnicu ● Bianca Stan ● Vali O.' isn't supported. A folder name must not start or end with '.', must not contain any unprintable characters or contain any of the following characters: '\', '/', ':', ';', '*', '<', '>', '|', '?'. (request_parameter_invalid)

xybu commented 9 years ago

Ignore list not working?

Sincerely, Xiangyu Bu

Sent from my Windows Phone

From: Emanuel Posescumailto:notifications@github.com Sent: ‎3/‎23/‎2015 3:32 AM To: xybu/onedrive-dmailto:onedrive-d@noreply.github.com Cc: Xiangyu Bumailto:xybu92@live.com Subject: Re: [onedrive-d] Syncing large number of files (#144)

I have another exception that crashes the worker (I will rename the folder)

Exception in thread worker2: Traceback (most recent call last): File "/usr/lib/python3.3/threading.py", line 637, in _bootstrap_inner self.run() File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 443, in run self.make_remote_dir(task) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 338, in make_remote_dir new_entry = self.api.mkdir(name, task['remote_parent_id']) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 328, in mkdir return self.parse_response(r, OneDriveAPIException, requests.codes.created) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 115, in parse_response raise error(ret) onedrive_d.od_onedriveapi.OneDriveAPIException: The operation couldn't be completed because the requested folder name, 'Hangout Bogdan Postelnicu ● Bianca Stan ● Vali O.' isn't supported. A folder name must not start or end with '.', must not contain any unprintable characters or contain any of the following characters: '\', '/', ':', ';', '*', '<', '>', '|', '?'. (request_parameter_invalid)

Reply to this email directly or view it on GitHub: https://github.com/xybu/onedrive-d/issues/144#issuecomment-84856893

kind3r commented 9 years ago

Ignore list is working for the extra content that I added, but there is no ignore rule (as far as I can tell) for folders ending in '.' (dot), just for those begining with '.'

xybu commented 9 years ago

Thanks for catching this. It is a bug. The server response is not documented in Live API either. Should issue a fix. But can't guarantee when. Floods of homework coming...

starrwang commented 9 years ago

If we could check if we've reached the limit on how many requests can be made in a period of time (some exception caught?), we can 'pause' the producer threads for a while. Or we can take some other rate control methords such as token bucket.

Might the mkdir conflict due to inconsistency? One worker has just made remote modification under the parent path while another just goes to scan same path. (remote change has not been observed yet?)

Also there is possibility that sync task for a path has returned while network IO under this path is on-going. I added another post on #153

xybu commented 9 years ago

@starrwang Bandwidth sounds like another topic. Please create a new issue for discussion. Not sure what kind of bandwidth you are talking about; not to mention how to detect it.

Also each local path can have at most one task (pending or running). This also holds for the new version.

starrwang commented 9 years ago

@xybu Sorry for unclearity, post was updated.