search

cjnaz / rclonesync-V2

A Bidirectional Cloud Sync Utility using rclone
MIT License
355 stars 39 forks source link

[Dropbox] Rclonesync Referencing files outside of filters and hanging #55

Closed Cralex closed 4 years ago

Cralex commented 4 years ago

I'm trying to set up a sync with dropbox on a laptop running Manjaro ARM. (DB really needs an ARM client, already...) The first sync runs fine, but every time I try to sync a second time I see a reference to some random file outside of my specified Filters file and a dump of html from the dropbox website. It then appears to hang in the terminal until I control-c to cancel the command. I'm attaching a copy of my filters and the log.

Filters and Logs.zip

The long contains the content from one successful first sync and the failed second sync. No files were changed locally or in the cloud between syncs. As you can see from my filters, I've been trying to specifically exclude some of the directories containing files that have appeared in the log in previous attempts, but at this rate I'll have to blacklist everything that I'm not already whitelisting. Maybe I'm not doing something right? Let me know if there's anything else you need from me to figure out what's going on. Thanks.

cjnaz commented 4 years ago

The second run is showing an error from rclone itself (based on the '/' characters in the date string). 

2020/07/08 07:28:37 ERROR : Minecraft Saves/saves/Note Block jae1000j: error listing: <!DOCTYPE html>
<html>

This file starts with a '!'. I'm not sure if/why this is a problem for rclone, and I don't understand why the contents of the file are being printed. Your filters are blocking whole sub-trees. Are you finding that it is choking on filenames with special characters, such as '!'?

Try running rlconesync with a double verbose switch to turn on the maximum logging --verbose --verbose, and also turn on rclone's verbose with --rc-verbose --rc-verbose.

Cralex commented 4 years ago

I don’t recall using any files with special characters in their file names, unless certain apps did so automatically when connected to Dropbox. Regardless, I cleaned up my Dropbox a little (removed as many subtrees as possible that contain files that are called out in the log) and ran first and second syncs with maximum verbosity: https://www.dropbox.com/s/6aixj3gmn27q9sd/dropbox_runlog?dl=0 This time, the job failed on its own without needing to be terminated. (Feel free time delete the link after you download the logs.)

cjnaz commented 4 years ago

It's failing at the Check Access stage. Note that your filters file is not used at this stage.

I placed a file named "!DOCTYPE html" on the root of my Dropbox folder (not with html content, not that that should matter, although your log looks like its dumping the content of the file) and ran this LSL, which ran as expected (only listing RCLONE_TEST files):

rclone lsl Dropbox: --filter "- /testdir/" --filter "- rclonesync/Test/" --filter "+ RCLONE_TEST" --filter "- *"

Hum... Please try these things:

From your log where it blows up...

2020-07-08 16:32:51,570:  >>>>> Checking Path1 and Path2 rclone filesystems access health
2020-07-08 16:32:51,587:      rclone command:  ['rclone', 'lsl', '/home/cralex/Dropbox/', '--config', '/home/cralex/.config/rclone/rclone.conf', '--filter', '- /testdir/', '--filter', '- rclonesync/Test/', '--filter', '+ RCLONE_TEST', '--filter', '- *']
2020-07-08 16:32:51,777:      rclone command:  ['rclone', 'lsl', 'Dropbox:', '--config', '/home/cralex/.config/rclone/rclone.conf', '--filter', '- /testdir/', '--filter', '- rclonesync/Test/', '--filter', '+ RCLONE_TEST', '--filter', '- *']
2020/07/08 16:33:35 ERROR : Apps/3DS Sync/checkpoint/saves/0x0EB80 Super Mario Land 2 DX/20180526-203357: error listing: <!DOCTYPE html>
<html>

Go ahead and delete your log file since it has lots of local path info that you may not want public. I've downloaded a copy from your dropbox, which I'll delete once we have this figured out.

Cralex commented 4 years ago

i just don't understand the !DOCTYPE html thing, since I searched my dropbox for DOCTYPE and didn't find any matching files. Anyway...

Also, does the default filter block all files that begin with a period? The initial sync pulled down some folders within my intended path that began with a period, but further changes to them don't appear to be synced back.

Edit: I tried running the lsl command again after some successful syncs (The first time I ran it was after the last failed sync yesterday.) and it did another dump in my terminal. I tried scrolling up to see where it began, but this is as high as it would go: output.txt

cjnaz commented 4 years ago

I think that the !DOCTYPE html output is the stderr stream being echoed by rclone when it gets an error response from Dropbox. I trimmed your latter dropbox_runlog and renamed to .html and it outputs as below. Again, this is from the checkaccess stage where the only filename that should be returned is RCLONE_TEST. I created the full path and filename Apps/3DS Sync/checkpoint/saves/0x0EB80 Super Mario Land 2 DX\20180526-203357 on my Dropbox and ran the rclone lsl command 10 times and it only returned my RCLONE_TEST files and no errors.

I suspect that either 1) rclone 1.52.2 has a bug (although I can't reproduce it), or 2) Dropbox is unhappy (maybe a bug).

Ideas and next steps 1) Do you have another sync agent running in parallel with rclonesync runs, perhaps the Dropbox agent on Windows? If yes, you can't run both. That might explain a file not found all the sudden.
2) Do you have a massive number of files on Dropbox? It's possible that you are hitting some type of quota and Dropbox is giving a misleading error. 3) Do the lsl fails consistently show the same file, or do they vary? If they vary then its not a individual file causing the problem. 4) If you are able to issue the above rclone lsl command AND get the error response then the problem is not with rclonesync. Post a new issue at https://github.com/rclone/rclone/issues. Also try installing/running an older version of rclone from https://downloads.rclone.org/ to see if it's a newer problem.

image

Cralex commented 4 years ago
  1. I have the official dropbox client running on another computer running Windows. It shouldn't be doing anything at the moment, if it matters.
  2. I do have a Calibre library of ebooks in dropbox. (It's in the Apps directory, which kept appearing in the error logs despite being filtered out.) There's about 10,000 files in the library, and about 15,800 files in my dropbox overall, totaling about 7 GB. For rclonesync, though, I'm trying to keep it to only the files I'll need.
  3. It seems to be different files, although they seem to generally be from a couple of general places in my files.

I'm debating just using rclone's mount feature instead of trying to sync, as I'm not going to be using this computer anywhere without internet and I probably won't need "instant" access to anything. That or moving what I want to sync onto another provider or something.

cjnaz commented 4 years ago

A mount is likely a good solution.

You exclude your Calibre library in your filters, yet the errors are coming from there during the check access phase, which suggests that the check access hard coded filters may not right. Check access needs to scan all directories, but shouldn't be picking up and erroring on any of your files. I could change the check access to only look for the check file in the top dir. This is unsafe, probably, if there are symlinked directories that may have dropped offline (resulting on deleting files on the other path).

I'll have a careful look at the check access hard coded filters (listed in the rclone lsl above, fyi).

(I need to add a note on the importance of --check-access files when using symlinked dirs.)

cjnaz commented 4 years ago

fyi, I put up about 20,000 files total in 100 different directories (200 files in each), and ran several rclone lsl commands as well as rclonesync runs with no errors. The files are zero bytes in size. I'm on rclone 1.52.2 on Centos 7.8.

This command runs in about 7 sec:

$ date; rclone lsl Dropbox: --filter "- /testdir/" --filter "- rclonesync/Test/" --filter "+ RCLONE_TEST" --filter "- *"; date
Fri Jul 10 19:15:01 MST 2020
      110 2017-07-28 23:19:06.000000000 RCLONE_TEST
      110 2017-07-28 23:19:07.000000000 cjnFiles/RCLONE_TEST
Fri Jul 10 19:15:08 MST 2020

rcdropbox is my alias...

$ rcdropbox 
2020-07-10 19:21:36,727:  ***** BiDirectional Sync for Cloud Services using rclone (V2.10 200411) *****
2020-07-10 19:21:36,761:  Lock file created: </tmp/rclonesync_LOCK__mnt_share_public_DBox_Dropbox_Dropbox_>
2020-07-10 19:21:36,762:  Synching Path1  </mnt/share/public/DBox/Dropbox/>  with Path2  <Dropbox:>
2020-07-10 19:21:36,762:  Command args: <Path1=/mnt/share/public/DBox/Dropbox, Path2=Dropbox:, check_access=True, check_filename=RCLONE_TEST, config=None, dry_run=False, filters_file=/home/cjn/.rclonesyncwd/Filters, first_sync=False, force=False, max_deletes=50, no_datetime_log=False, rc_verbose=None, rclone=rclone, rclone_args=None, remove_empty_directories=False, verbose=1, workdir=/home/cjn/.rclonesyncwd>
2020-07-10 19:21:36,762:  Using filters-file  </home/cjn/.rclonesyncwd/Filters>
2020-07-10 19:21:36,762:  >>>>> Checking Path1 and Path2 rclone filesystems access health
2020-07-10 19:21:51,159:  >>>>> Path1 Checking for Diffs
2020-07-10 19:21:51,166:  >>>>> Path2 Checking for Diffs
2020-07-10 19:21:51,173:  >>>>> No changes on Path2 - Skipping ahead
2020-07-10 19:21:51,173:  >>>>> No changes on Path1 or Path2 - Skipping sync from Path1 to Path2
2020-07-10 19:21:51,173:  >>>>> Refreshing Path1 and Path2 lsl files
2020-07-10 19:21:58,175:  Lock file removed: </tmp/rclonesync_LOCK__mnt_share_public_DBox_Dropbox_Dropbox_>
2020-07-10 19:21:58,175:  >>>>> Successful run.  All done.

What do you suggest may be worse case? How many files do you have in the directory that has errors?

Can you confirm that the above rlcone lsl Dropbox: command sometimes gives you the error when run by itself?

cjnaz commented 4 years ago

fyi, V2.11 to be posted soon has reworked filtering for check access phase, which should fix this problem. Interested in testing it out?

cjnaz commented 4 years ago

Fixed in v2.11