As @miyunari proposed, we should evaluate the usage of zstandard compressed files as delivery method of raw image data.
Currently the retriever creates files that are zst encoded (AWS) as well as non encoded files (google and azure).
This gives the impression that the question of encoding raw image data isn't well enough thought out.
Expected Outcome
If the decision is made to transition to zst encoding, then the transformer needs to be extended to enable decompression of said files.
Implementation example: https://github.com/redhatcloudx/transformer/pull/574
It also requires that the retriever creates consistent data across the board.
All created raw data files need to be zst encoded, not only AWS
As @miyunari proposed, we should evaluate the usage of zstandard compressed files as delivery method of raw image data. Currently the retriever creates files that are zst encoded (AWS) as well as non encoded files (google and azure). This gives the impression that the question of encoding raw image data isn't well enough thought out.
Expected Outcome If the decision is made to transition to zst encoding, then the transformer needs to be extended to enable decompression of said files. Implementation example: https://github.com/redhatcloudx/transformer/pull/574
It also requires that the retriever creates consistent data across the board.