search

numblr / glaciertools

Command line (bash) scripts to upload large files to AWS glacier using multipart upload and to calculate the required tree hash
MIT License
69 stars 19 forks source link

Improve performance related to variables holding part data #7

Closed numblr closed 6 years ago

numblr commented 6 years ago

To implement the part upload chunks of the file are kept in shell variables and processed using echo -n (mainly to determine the range of the chunk). It seems to be more efficient to store those large variables to a temporary file, see e.g. here: https://superuser.com/questions/279141/why-is-reading-a-file-faster-than-reading-a-variable

An alternative mechanism to determine the range might be an other option.

numblr commented 6 years ago

The sequence number of the part can be retrieved from a replacement string from in the command argument in parallel, which makes hex conversion, adding line numbers and the related storage of the data in shell variables obsolete.

numblr commented 6 years ago

Merged in #8.