I think when we ere developing this we were using a file with a ".tar" extension rather than a file that was compressed as a tar. this means then when I tried to decompress an actual tar file it failed with an encoding error.
I've just explicitly switched the logic so we:
download the tar file to a temporary file
untar it from there
because we're no longer using download_s3_content_to_local we don't get tripped by the ""utf-8"" assumption.
How to review
Try it out. Will put a sample s3 object elsewhere (not in a public github repo)
from dpytools.s3.basic import decompress_s3_tar
decompress_s3_tar("sample d3 object", "stuff")
Note: I've only actually changed about 3 lines in the function, the tests and documentation remain as is. Most of the changes are from running make fmt and make lint.
What
Quick fix for the s3 decompress function.
I think when we ere developing this we were using a file with a ".tar" extension rather than a file that was compressed as a tar. this means then when I tried to decompress an actual tar file it failed with an encoding error.
I've just explicitly switched the logic so we:
because we're no longer using
download_s3_content_to_localwe don't get tripped by the ""utf-8"" assumption.How to review
Try it out. Will put a sample s3 object elsewhere (not in a public github repo)
Note: I've only actually changed about 3 lines in the function, the tests and documentation remain as is. Most of the changes are from running
make fmtandmake lint.Who can review
Anyone.