IndexError: list index out of range when continuing a sync started with Cyberduck

[briancsimonsen]

The error occurs while trying to sync a large number of files (~139000) for upload. The process worked correctly on default settings up until nearly the end where the following error occurred. This sync has never successfully completed and is going to a new bucket on a new account.

H:\>b2 sync --skipNewer . b2://FamilyLaCieBackup/Cyberduck
ERROR:b2sdk.sync.action:an exception occurred in a sync action
Traceback (most recent call last):
  File "c:\program files\python37\lib\site-packages\b2sdk\sync\action.py", line 49, in run
    self.do_action(bucket, reporter)
  File "c:\program files\python37\lib\site-packages\b2sdk\sync\action.py", line 129, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "c:\program files\python37\lib\site-packages\logfury\v0_1\trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "c:\program files\python37\lib\site-packages\b2sdk\bucket.py", line 591, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "c:\program files\python37\lib\site-packages\b2sdk\bucket.py", line 649, in _upload_large_file
    part_ranges,
  File "c:\program files\python37\lib\site-packages\b2sdk\bucket.py", line 696, in _find_unfinished_file_if_possible
    offset, part_length = part_ranges[part.part_number - 1]
IndexError: list index out of range
b2_upload(\\?\H:\mp4FILES\REEL14.mp4, Cyberduck/mp4FILES/REEL14.mp4, 1369161244000): IndexError('list index out of range') list index out of range
ERROR:b2sdk.sync.action:an exception occurred in a sync action
Traceback (most recent call last):
  File "c:\program files\python37\lib\site-packages\b2sdk\sync\action.py", line 49, in run
    self.do_action(bucket, reporter)
  File "c:\program files\python37\lib\site-packages\b2sdk\sync\action.py", line 129, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "c:\program files\python37\lib\site-packages\logfury\v0_1\trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "c:\program files\python37\lib\site-packages\b2sdk\bucket.py", line 591, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "c:\program files\python37\lib\site-packages\b2sdk\bucket.py", line 649, in _upload_large_file
    part_ranges,
  File "c:\program files\python37\lib\site-packages\b2sdk\bucket.py", line 696, in _find_unfinished_file_if_possible
    offset, part_length = part_ranges[part.part_number - 1]
IndexError: list index out of range
b2_upload(\\?\H:\mp4FILES\REEL5.mp4, Cyberduck/mp4FILES/REEL5.mp4, 1369151190000): IndexError('list index out of range') list index out of range
ERROR:b2.console_tool:ConsoleTool command error
Traceback (most recent call last):
  File "c:\program files\python37\lib\site-packages\b2\console_tool.py", line 1361, in run_command
    return command.run(args)
  File "c:\program files\python37\lib\site-packages\b2\console_tool.py", line 1184, in run
    allow_empty_source=allow_empty_source
  File "c:\program files\python37\lib\site-packages\logfury\v0_1\trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "c:\program files\python37\lib\site-packages\b2sdk\sync\sync.py", line 273, in sync_folders
    raise CommandError('sync is incomplete')
b2sdk.exception.CommandError: sync is incomplete
ERROR: sync is incomplete

This is a copy paste from the last try but I have tried a few different things to get around this error.

• Python 2.7 (32 bit and 64 bit)
• Change number of threads (1, 10, 100)
• Python 3.7 (64 bit)
• reinstall the b2 CLI

[svonohr]

I think this is a bug in B2 CLI. The _find_unfinished_file_if_possible function assumes that the number of parts on B2 is at most equal to the number of parts for the local file. If there are more parts on B2 it will probably fail with this error you are seeing (haven't reproduced it).

Did you change the part size during your sync or did the file size of the corresponding files change? Until this bug is resolved you can delete the unfinished file from B2 and try syncing again.

[briancsimonsen]

The files are all for archival purposes so none of the files should be edited in any way. However, the upload was originally attempted using Cyberduck but the speed was so slow that I switched to the CLI. I don't think that would be the cause considering the command I am running uses the default filename and modification time compare so all partially uploaded files would be skipped.

[ppolewicz]

If Cyberduck used a different part size than CLI, then it could be a problem.

In order to complete the sync, you could cancel the large file uploads (easy, via b2 cancel-all-unfinished-large-files <bucketName> or b2 list-unfinished-large-files <bucketName> and b2 cancel-large-file <fileId>) or try to match the part size with whatever cyberduck chose (hard).

[briancsimonsen]

@ppolewicz was correct. There were quite a few unfinished files. After running the b2 cancel-all-unfinished-large-files <bucketName> command everything started working as expected and uploaded successfully.

[ppolewicz]

We should provide a useful error message instead of the current behavior, but I don't think we should implement a mechanism to detect the part size of the interrupted large file upload. It is possible, but it would never get used and the workaround is trivial, so we just need to tell the user what it is.

[svonohr]

Why an error message? If there are more parts on B2 than local parts, the unfinished file is not a match and we continue similar to when the file info is different.

[ppolewicz]

Fair point

[briancsimonsen]

If unfinished files require system resources, treating unfinished files as unique and ignoring them might not be a good idea.