Revert a previous change that limited the size of the uploaded archive

uskudnik / amazon-glacier-cmd-interface

Command line interface for Amazon Glacier

MIT License

374 stars 100 forks source link

Revert a previous change that limited the size of the uploaded archive #35

Closed gburca closed 12 years ago

gburca commented 12 years ago

Making the part_size smaller in the GlacierWriter constructor has the side-effect of limiting the largest archive size that can be uploaded since AWS limits the upload to 10,000 parts of that size. Need to find a different way of providing progress indications.

uskudnik commented 12 years ago

@gburca Awesome, I didn't even know about 10 000 parts of that size limit.

And I'm joining @wvmarle in appreciation of progress indicator :)

gburca commented 12 years ago

Sorry to break it to you @wvmarle, but if your upload failed at 9-20Gb, this patch probably won't fix that. This patch fixes ULs that fail after exactly 4_1024_1024*10000 bytes (40Gb). Anything less than 40Gb should have worked just fine.

As for the "silent" mode, the progress is only shown if the script is attached to a TTY. The rest of the output is shown at all times.

I agree that a % indicator would be useful. It would only work when the full archive size is known in advance (i.e. it won't work if the script is reading the archive contents from a pipe or from STDIN).

wvmarle commented 12 years ago

I don't know which patch fixed it, but it seems to be working now :-)

So now it's going to fail after 1,280 GB?

Not many people have files larger than this size, it's still something that may have to be documented at least. Or fixed: if file size >1280 GB recalculate part size.

gburca commented 12 years ago

I just added the --partsize option to the upload subcommand so the user can specify the size to be used in case the default is not adequate. See: https://github.com/gburca/amazon-glacier-cmd-interface/commit/e243a9a8e534f9ef539227702139a69cc56ca212