redhatcloudx / transformer

Find RHEL images in public clouds
Apache License 2.0
6 stars 3 forks source link

Evaluate usage of zst compressed files for the raw image data #575

Open F-X64 opened 1 year ago

F-X64 commented 1 year ago

As @miyunari proposed, we should evaluate the usage of zstandard compressed files as delivery method of raw image data. Currently the retriever creates files that are zst encoded (AWS) as well as non encoded files (google and azure). This gives the impression that the question of encoding raw image data isn't well enough thought out.

Expected Outcome If the decision is made to transition to zst encoding, then the transformer needs to be extended to enable decompression of said files. Implementation example: https://github.com/redhatcloudx/transformer/pull/574

It also requires that the retriever creates consistent data across the board.

F-X64 commented 1 year ago

As discussed on June 5th we find the zstandard encryption to be a valuable addition. Implementation will be postponed until Q3.