Bundles built with Python3.8 are not formatted properly

colcon / colcon-bundle

A colcon extension to create portable application bundles

Apache License 2.0

38 stars 24 forks source link

Bundles built with Python3.8 are not formatted properly #160

Closed murphm8 closed 4 years ago

murphm8 commented 4 years ago

A while ago I made a fix for a Python3.8 test failure: #86

There still appears to be some sort of problem with bundles build with Python3.8. When I run the bundle extraction tool on it (uses the metadata), I see the following:

go run github.com/aws-robotics/aws-robomaker-bundle-support-library/cmd/cli  --bundle ~/output.tar 
Processing overlay: {FileName:dependencies.tar.gz Sha256:7cfe6853455b6dd6b3d88c1680272c503170c4077282e0f1f76e105bc654b5cd Offset:4195840 Size:1117671732}
2020/04/02 15:51:39 [EXTRACTION] bundle error: error decompressing: gzip: invalid header

prajakta-gokhale commented 4 years ago

This test failure is because tarfile.DEFAULT_FORMAT changed from GNU_FORMAT to PAX_FORMAT in Python 3.8.

With this change, the header size (and subsequently offset) of overlays calculated in bundlefile.py changed from 512 bytes in GNU_FORMAT to 1536 bytes in PAX_FORMAT.

For the unit tests to work in both python3.8, we need to account for the extra header size in unit tests.

prajakta-gokhale commented 4 years ago

There were 2 problems why tests were failing and why the header size was different in different python versions:

Default format is set to tarfile.GNU_FORMAT in close(), so all operations performed before (like initialization) had format mismatch.
Format is not passed as a parameter while calculating header and thus offset sizes.

Fix is in https://github.com/colcon/colcon-bundle/pull/161.