search

cjnaz / rclonesync-V2

A Bidirectional Cloud Sync Utility using rclone
MIT License
355 stars 39 forks source link

Dropbox non-ASCII character issue #35

Closed jefferyyjhsu closed 4 years ago

jefferyyjhsu commented 4 years ago

One of the folders that I sync between my local file system and Dropbox has non-ASCII characters. I used to use V2.4 and things worked fine until recently. It seems like Dropbox has changed the way they handle non-ASCII characters. To demonstrate the issue, assuming my folder is called XYZ (An actual working example would be 測試). The existing content in XYZ on my local machine and Dropbox is synced. Now, if I add a new file named test.txt to XYZ on my local file system. After syncing, a new folder with a garbled ASCII name containing test.txt will be created on Dropbox. I have tested the latest release from Oct 4th and it exhibits the same issue.

cjnaz commented 4 years ago

Thanks Jeffery - I did a few hours of experimenting and came to the conclusion that the problem is in rclone or Dropbox itself. Please see rclone issue #3609.

Questions... Are you seeing this intermittently, or 100% repeatably? Have you recently updated your rclone version? What platform are you running on?

I added your test characters to my test cases for future coverage.

jefferyyjhsu commented 4 years ago

Hi chnaz, Thanks for the quite response! I have tried to verify whether or not this issue is persistent today. However, it seems like with dropbox reverting the change that caused the issue, I can't reproduce it anymore. However, while doing my tests, it seems like if I create a new folder with the name 測試 on both dropbox and my machine, I will get the error messages shown below.

$ python rclonesync.py /tmp/測試/ dropbox:測試/ --first-sync

2019-10-11 09:02:31,876: BiDirectional Sync for Cloud Services using rclone 2019-10-11 09:02:32,005: Lock file created: </tmp/rclonesync_LOCKtmp_測試_dropbox測試_> 2019-10-11 09:02:32,006: Synching Path1 </tmp/測試/> with Path2 <dropbox:/測試/> 2019-10-11 09:02:32,006: Command line: <Namespace(Path1='/tmp/\xe6\xb8\xac\xe8\xa9\xa6/', > Path2='dropbox:\xe6\xb8\xac\xe8\xa9\xa6/', check_access=False, check_filename='RCLONE_TEST', > config=None, dry_run=False, filters_file=None, first_sync=True, force=False, max_deletes=50, no_datetime_log=False, rc_verbose=None, rclone='rclone', rclone_args=None, remove_empty_directories=False, verbose=False, workdir='home/jeffery/.rclonesyncwd')> 2019-10-11 09:02:32,007: >>>>> --first-sync copying any unique Path2 files to Path1 Traceback (most recent call last): File "rclonesync.py", line 733, in status = bidirSync() File "rclonesync.py", line 197, in bidirSync src = path2_base + key UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 9: ordinal not in range(128)

In summary:

  1. I can not reproduce the issue by adding new files to an existing folder on my machine and then sync it with dropbox.
  2. If I create a new folder with a non-ASCII character name, I consistently won't be able to sync it.
  3. A similar UnicodeDecodeError also shows up when I add new files to an existing folder on dropbox.
cjnaz commented 4 years ago

What is your platform (Linux it looks like), and what version of Python are you running?

jefferyyjhsu commented 4 years ago

Operating System: Debian GNU/Linux 9 (stretch) Kernel: Linux 4.19.66-v7+ Architecture: arm

$ python Python 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516] on linux2

cjnaz commented 4 years ago

The issue is with entering non-ASCII paths on the command line. This affects Path1, Path2, --workdir, --rclone, --check-filename, and --filters-file. I've identified solutions that should work on Python 2.7 on Linux and Windows, and on Linux 3.x. It looks like for Windows py2.7 support we'll need another module that handles utf-8 command line entry. I'll get to these changes later next week, fyi. As a work around, just make the path1/path2 params have ASCII only, or py 3.x may work as is.

cjnaz commented 4 years ago

Should be resolved in V2.9. Please check it out.

cjnaz commented 4 years ago

Closing

jefferyyjhsu commented 4 years ago

Thanks @cjnaz. Sorry that I missed your last comment from a week ago.

cjnaz commented 4 years ago

Please give it a try

On Tue, Nov 12, 2019, 8:24 AM jefferyyjhsu notifications@github.com wrote:

Thanks @cjnaz https://github.com/cjnaz. Sorry that I missed your last comment from a week ago.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cjnaz/rclonesync-V2/issues/35?email_source=notifications&email_token=ADRKVYIR3I2H6HU56FX2JATQTLDCJA5CNFSM4I62BVTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED2TNZA#issuecomment-552941284, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRKVYM7KNWN5GIKDNFA77LQTLDCJANCNFSM4I62BVTA .

jefferyyjhsu commented 4 years ago

I have tested in the past few days. The latest version has performed flawlessly. Thanks, @cjnaz !