When uploading without a size hint (via headers or --size-hint), a x-archive-size-hint header is added automatically. However, prior to this commit, the value was the individual file size on each file's PUT request. This effectively made the size hint useless because it does not, in fact, provide a size hint for the total item size to the S3 backend at item creation time.
Notes on detailed changes to implement this:
Rename internetarchive.utils.recursive_file_count to recursive_file_count_and_size, adding a wrapper for backwards compatibility
Add support for paths (rather than only file-like objects) to internetarchive.utils.get_file_size
Add a internetarchive.utils.is_filelike_obj helper function
Fix a bug introduced by 62c85133090e21659f09964fedd4c18ae4a27483 where total_files would never be None and so recursive_file_count was never called, possibly leading to incorrect derive queueing.
When uploading without a size hint (via headers or
--size-hint
), a x-archive-size-hint header is added automatically. However, prior to this commit, the value was the individual file size on each file'sPUT
request. This effectively made the size hint useless because it does not, in fact, provide a size hint for the total item size to the S3 backend at item creation time.Notes on detailed changes to implement this:
internetarchive.utils.recursive_file_count
torecursive_file_count_and_size
, adding a wrapper for backwards compatibilityinternetarchive.utils.get_file_size
internetarchive.utils.is_filelike_obj
helper functiontotal_files
would never beNone
and sorecursive_file_count
was never called, possibly leading to incorrect derive queueing.