search

yadayada / acd_cli

An unmaintained command line interface and FUSE filesystem for Amazon (Cloud) Drive
Other
1.35k stars 165 forks source link

Name collision for some Unicode filenames #546

Open Snack-X opened 7 years ago

Snack-X commented 7 years ago

I started to get error messages like Name collision with non-cached file. If you want to overwrite, please sync and try again. when I add more files and try to upload them with -o options,

Uploads end successfully, but this looks like some kind of error or bug. Expected behavior is no errors at all.

First, I install from master branch, and my system looks like this:

$ pip3 install --upgrade git+https://github.com/yadayada/acd_cli.git
Collecting git+https://github.com/yadayada/acd_cli.git
(...skipped...)
Successfully installed acdcli-0.3.2 appdirs-1.4.3

$ acd_cli version
acd_cli 0.3.2, api 0.9.2

$ python3 -c 'import platform as p; print("%s\n%s" % (p.python_version(), p.platform()))'
3.5.2
Linux-4.4.0-64-generic-x86_64-with-Ubuntu-16.04-xenial

Then I clear cache and sync.

$ acd_cli clear-cache 

$ acd_cli sync
Getting changes.............
Inserting nodes............

And, upload. At the first time, upload was successful so there are files and some of the files duplicate. There are more files which produces error, but this is one of them:

$ acd_cli -v upload -o "FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver\!\! [FLAC
]" /Musics
(...skipped...)
17-03-30 22:45:19.496 [INFO] [acdcli.utils.threading] - 8 jobs in queue.
17-03-30 22:45:19.499 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/01. ØωØver!!.flac
17-03-30 22:45:19.504 [INFO] [acdcli.api.backoff_req] - POST "https://content-na.drive.amazonaws.com/cdproxy/nodes"
17-03-30 22:45:34.297 [ERROR] [acd_cli] - Uploading "01. ØωØver!!.flac" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:34.301 [INFO] [acd_cli] - Skipping upload of "02. 夕映えプレゼント -(Asterisk)リミックス-.flac" because of mtime or ctime and size.
17-03-30 22:45:34.302 [INFO] [acd_cli] - Skipping upload of "03. ボナスドラマ「マジックアワ(Asterisk)特別編」.flac" because of mtime or ctime and size.
17-03-30 22:45:34.303 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/04. ØωØver!! (オリジナルカラオケ).flac
17-03-30 22:45:47.480 [ERROR] [acd_cli] - Uploading "04. ØωØver!! (オリジナルカラオケ).flac" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:47.481 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.cue
17-03-30 22:45:51.148 [ERROR] [acd_cli] - Uploading "THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.cue" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:51.148 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.log
17-03-30 22:45:52.664 [ERROR] [acd_cli] - Uploading "THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.log" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:52.665 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.m3u
17-03-30 22:45:55.672 [ERROR] [acd_cli] - Uploading "THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.m3u" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:55.673 [INFO] [acd_cli] - Skipping upload of "cover.jpg" because of mtime or ctime and size.
[#########################] 100.0% of  188MiB  8/8  130.7KB/s       0s
5 file(s) failed.
17-03-30 22:45:55.912 [WARNING] [acd_cli] - Return value error code: 256.

But as I mentioned before, files are there.

$ acd_cli ls "/Musics/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver\!\! [FLAC]"
[o6HbM7xrSJSzABbxGNBbig] [A] 01. ØωØver!!.flac
[kQJ6Pb0nS1qz4rVTQ0dAWg] [A] 02. 夕映えプレゼント -(Asterisk)リミックス-.flac
[LT06psrpRHSHBDVTWmseWA] [A] 03. ボナスドラマ「マジックアワ(Asterisk)特別編」.flac
[5ny0WF_nQV-9lgL2ubVjGw] [A] 04. ØωØver!! (オリジナルカラオケ).flac
[LuLZreLURQa5rbmARb0FUA] [A] THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.cue
[WwjfSBFLSbqekyLZEGNyBA] [A] THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.log
[kZZNrcluTZy20ikwi4HNhg] [A] THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.m3u
[BNmGHYsuR_iDIp-8B79yKQ] [A] cover.jpg

$ ls "FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver\!\! [FLAC]"
01. ØωØver!!.flac
02. 夕映えプレゼント -(Asterisk)リミックス-.flac
03. ボナスドラマ「マジックアワ(Asterisk)特別編」.flac
04. ØωØver!! (オリジナルカラオケ).flac
cover.jpg
THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.cue
THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.log
THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.m3u

$ md5sum "FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver\!\! [FLAC]/01. ØωØver\!\!.flac"
caa6986d4e579da9c745d8e30bb889f3  FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/01. ØωØver!!.flac

$ acd_cli find-md5 "caa6986d4e579da9c745d8e30bb889f3"
[o6HbM7xrSJSzABbxGNBbig] [A] /Musics/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/01. ØωØver!!.flac

I thought it was some kind of Unicode encoding error, but filenames are matching:

$ acd_cli ls "/Musics/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver\!\! [FLAC]" | grep 01 | tail -c 21 | xxd
00000000: 3031 2e20 c398 cf89 c398 7665 7221 212e  01. ......ver!!.
00000010: 666c 6163 0a                             flac.

$ ls "FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver\!\! [FLAC]" | grep 01 | tail -c 21 | xxd
00000000: 3031 2e20 c398 cf89 c398 7665 7221 212e  01. ......ver!!.
00000010: 666c 6163 0a                             flac.

Not all Unicode filenames are failing, but some does.

Below are logs:

17-03-30 22:45:19.353 [INFO] [acd_cli] - Plugin leaf classes: TestPlugin
17-03-30 22:45:19.354 [INFO] [acd_cli] - TestPlugin attached.
17-03-30 22:45:19.354 [INFO] [acd_cli] - Settings path is "/root/.config/acd_cli".
17-03-30 22:45:19.355 [INFO] [acd_cli] - Cache path is "/root/.cache/acd_cli".
17-03-30 22:45:19.356 [DEBUG] [acdcli.utils.conf] - configuration resulting from merging default and acd_cli.ini: {'download': {'keep_corrupt': 'False', 'keep_incomplete': 'True'}, 'DEFAULT': {}, 'upload': {'timeout_wait': '10'}}
17-03-30 22:45:19.357 [DEBUG] [acdcli.utils.conf] - configuration resulting from merging default and acd_client.ini: {'DEFAULT': {}, 'proxies': {}, 'transfer': {'idle_timeout': '60', 'fs_chunk_size': '131072', 'dl_chunk_size': '524288000', 'connection_timeout': '30', 'chunk_retries': '1'}, 'endpoints': {'filename': 'endpoint_data', 'validity_duration': '259200'}}
17-03-30 22:45:19.357 [INFO] [acdcli.api.client] - Initializing ACD with path "/root/.cache/acd_cli".
17-03-30 22:45:19.358 [INFO] [acdcli.api.oauth] - AppspotOAuthHandler initialized
17-03-30 22:45:19.361 [DEBUG] [acdcli.utils.conf] - configuration resulting from merging default and cache.ini: {'DEFAULT': {}, 'blacklist': {'folders': '[]'}, 'sqlite': {'filename': 'nodes.db', 'journal_mode': 'wal', 'busy_timeout': '30000'}}
17-03-30 22:45:19.490 [INFO] [acdcli.cache.schema] - DB schema version is 2.
17-03-30 22:45:19.491 [DEBUG] [acdcli.cache.db] - Set busy_timeout to 30000. Result: 30000.
17-03-30 22:45:19.491 [DEBUG] [acdcli.cache.db] - Set journal_mode to wal. Result: wal.
17-03-30 22:45:19.492 [INFO] [acd_cli] - Last sync at 2017-03-30 13:45:00.033880.
17-03-30 22:45:19.493 [INFO] [acd_cli] - Resolved "/Musics" to "Node('u2056M5HQPeKMf4jcKGkfQ', 'Musics')"
17-03-30 22:45:19.493 [DEBUG] [acd_cli] - Namespace(acd_client=<acdcli.api.client.ACDClient object at 0x7f9bf87f9ba8>, action='upload', cache=<acdcli.cache.db.NodeCache object at 0x7f9bf87c2b38>, check=0, color=2, debug=None, deduplicate=False, exclude_fe=[], exclude_path=[], exclude_re=[], force=False, func=<function upload_action at 0x7f9bf882fa60>, log=True, max_connections=1, max_retries=0, no_wait=False, overwrite=True, parent='u2056M5HQPeKMf4jcKGkfQ', parent_path='/Musics', path=['FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]'], print_progress=True, remove_source_files=False, utf=False, verbose=1)
17-03-30 22:45:19.496 [INFO] [acdcli.utils.threading] - 8 jobs in queue.
17-03-30 22:45:19.499 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/01. ØωØver!!.flac
17-03-30 22:45:19.503 [DEBUG] [acdcli.api.backoff_req] - Retry 0, waiting -0.143509s
17-03-30 22:45:19.504 [INFO] [acdcli.api.backoff_req] - POST "https://content-na.drive.amazonaws.com/cdproxy/nodes"
17-03-30 22:45:19.504 [DEBUG] [acdcli.api.backoff_req] - <MultipartEncoder: OrderedDict([('metadata', '{"parents": ["pYkDeQQDSyuDnNTnTfZ2Ug"], "name": "01. \\u00d8\\u03c9\\u00d8ver!!.flac", "kind": "FILE"}'), ('content', ('filename', <acdcli.api.content._TeeBufferedReader object at 0x7f9bf87cc588>, 'audio/flac'))])>
17-03-30 22:45:19.507 [DEBUG] [requests.packages.urllib3.connectionpool] - Starting new HTTPS connection (1): content-na.drive.amazonaws.com
17-03-30 22:45:34.263 [DEBUG] [requests.packages.urllib3.connectionpool] - https://content-na.drive.amazonaws.com:443 "POST /cdproxy/nodes?suppress=deduplication HTTP/1.1" 409 269
17-03-30 22:45:34.297 [ERROR] [acd_cli] - Uploading "01. ØωØver!!.flac" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:34.301 [DEBUG] [acd_cli] - Remote mtime: 2017-03-10 10:52:15.272000, local mtime: 2015-04-16 12:33:05, local ctime: 2016-08-27 05:04:27.919096
17-03-30 22:45:34.301 [INFO] [acd_cli] - Skipping upload of "02. 夕映えプレゼント -(Asterisk)リミックス-.flac" because of mtime or ctime and size.
17-03-30 22:45:34.302 [DEBUG] [acd_cli] - Remote mtime: 2017-03-10 10:52:39.301000, local mtime: 2015-04-16 12:33:05, local ctime: 2016-08-27 05:04:30.783105
17-03-30 22:45:34.302 [INFO] [acd_cli] - Skipping upload of "03. ボナスドラマ「マジックアワ(Asterisk)特別編」.flac" because of mtime or ctime and size.
17-03-30 22:45:34.303 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/04. ØωØver!! (オリジナルカラオケ).flac
17-03-30 22:45:34.303 [DEBUG] [acdcli.api.backoff_req] - Retry 1, waiting 1.496020s
17-03-30 22:45:35.802 [DEBUG] [acdcli.api.backoff_req] - POST "https://content-na.drive.amazonaws.com/cdproxy/nodes"
17-03-30 22:45:35.802 [DEBUG] [acdcli.api.backoff_req] - <MultipartEncoder: OrderedDict([('metadata', '{"parents": ["pYkDeQQDSyuDnNTnTfZ2Ug"], "name": "04. \\u00d8\\u03c9\\u00d8ver!! (\\u30aa\\u30ea\\u30b8\\u30ca\\u30eb\\u30ab\\u30e9\\u30aa\\u30b1).flac", "kind": "FILE"}'), ('content', ('filename', <acdcli.api.content._TeeBufferedReader object at 0x7f9bf87e5b38>, 'audio/flac'))])>
17-03-30 22:45:47.472 [DEBUG] [requests.packages.urllib3.connectionpool] - https://content-na.drive.amazonaws.com:443 "POST /cdproxy/nodes?suppress=deduplication HTTP/1.1" 409 299
17-03-30 22:45:47.480 [ERROR] [acd_cli] - Uploading "04. ØωØver!! (オリジナルカラオケ).flac" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:47.481 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.cue
17-03-30 22:45:47.482 [DEBUG] [acdcli.api.backoff_req] - Retry 2, waiting 2.990608s
17-03-30 22:45:50.476 [DEBUG] [acdcli.api.backoff_req] - POST "https://content-na.drive.amazonaws.com/cdproxy/nodes"
17-03-30 22:45:50.477 [DEBUG] [acdcli.api.backoff_req] - <MultipartEncoder: OrderedDict([('metadata', '{"parents": ["pYkDeQQDSyuDnNTnTfZ2Ug"], "name": "THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 \\u00d8\\u03c9\\u00d8ver!!.cue", "kind": "FILE"}'), ('content', ('filename', <acdcli.api.content._TeeBufferedReader object at 0x7f9bf87e5dd8>, 'application/octet-stream'))])>
17-03-30 22:45:51.130 [DEBUG] [requests.packages.urllib3.connectionpool] - https://content-na.drive.amazonaws.com:443 "POST /cdproxy/nodes?suppress=deduplication HTTP/1.1" 409 317
17-03-30 22:45:51.148 [ERROR] [acd_cli] - Uploading "THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.cue" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:51.148 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.log
17-03-30 22:45:51.149 [DEBUG] [acdcli.api.backoff_req] - Retry 3, waiting 0.942298s
17-03-30 22:45:52.093 [DEBUG] [acdcli.api.backoff_req] - POST "https://content-na.drive.amazonaws.com/cdproxy/nodes"
17-03-30 22:45:52.093 [DEBUG] [acdcli.api.backoff_req] - <MultipartEncoder: OrderedDict([('metadata', '{"parents": ["pYkDeQQDSyuDnNTnTfZ2Ug"], "name": "THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 \\u00d8\\u03c9\\u00d8ver!!.log", "kind": "FILE"}'), ('content', ('filename', <acdcli.api.content._TeeBufferedReader object at 0x7f9bf87e5b38>, 'application/octet-stream'))])>
17-03-30 22:45:52.645 [DEBUG] [requests.packages.urllib3.connectionpool] - https://content-na.drive.amazonaws.com:443 "POST /cdproxy/nodes?suppress=deduplication HTTP/1.1" 409 317
17-03-30 22:45:52.664 [ERROR] [acd_cli] - Uploading "THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.log" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:52.665 [INFO] [acd_cli] - Uploading /mnt/hdd1/Rip/CD/FLAC/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!! [FLAC]/THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.m3u
17-03-30 22:45:52.666 [DEBUG] [acdcli.api.backoff_req] - Retry 4, waiting 2.548480s
17-03-30 22:45:55.218 [DEBUG] [acdcli.api.backoff_req] - POST "https://content-na.drive.amazonaws.com/cdproxy/nodes"
17-03-30 22:45:55.218 [DEBUG] [acdcli.api.backoff_req] - <MultipartEncoder: OrderedDict([('metadata', '{"parents": ["pYkDeQQDSyuDnNTnTfZ2Ug"], "name": "THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 \\u00d8\\u03c9\\u00d8ver!!.m3u", "kind": "FILE"}'), ('content', ('filename', <acdcli.api.content._TeeBufferedReader object at 0x7f9bf9b2ab38>, 'audio/x-mpegurl'))])>
17-03-30 22:45:55.651 [DEBUG] [requests.packages.urllib3.connectionpool] - https://content-na.drive.amazonaws.com:443 "POST /cdproxy/nodes?suppress=deduplication HTTP/1.1" 409 317
17-03-30 22:45:55.672 [ERROR] [acd_cli] - Uploading "THE IDOLM@STER CINDERELLA GIRLS ANIMATION PROJECT 06 ØωØver!!.m3u" failed. Name collision with non-cached file. If you want to overwrite, please sync and try again.
17-03-30 22:45:55.673 [DEBUG] [acd_cli] - Remote mtime: 2017-03-10 10:52:20.273000, local mtime: 2015-04-16 12:05:31, local ctime: 2016-08-27 05:04:31.739108
17-03-30 22:45:55.673 [INFO] [acd_cli] - Skipping upload of "cover.jpg" because of mtime or ctime and size.
17-03-30 22:45:55.912 [WARNING] [acd_cli] - Return value error code: 256.
yadayada commented 7 years ago

Can you please upgrade and have a look at the improved error messages?