Data missing when dumping gzip'd ion

nirosys commented 2 months ago

I have a gzip from one of our users that contains 2541 rows. When using the ion-cli to dump the data, it only produces 1213 rows, without error.

The issue seems to be with the gzip itself and how flate2 is decompressing it:

IonJava's CLI does not reproduce the issue.
Using IonJava to dump the data, then using MacOS's gzip to compress the data, ion-cli is able to process all of the data.
I built a quick rust app using flate2 to eliminate ion-cli entirely, and the same issue presents. Exactly 1213 rows are decompressed.

More investigation into what makes the archive different, and how we should properly handle it with flate2 is needed.

tgregg commented 2 months ago

It would be interesting to find out if the 1213 rows that are read successfully occur within the same compression frame, and if the remaining rows are contained in another frame (or frames) concatenated within the same file.

nirosys commented 2 months ago

Yep, that could definitely be it. The flate2 docs for GzDecoder say it can only handle a single member. Testing now with MultiGzDecoder.

nirosys commented 2 months ago

Fixed with #127, closing.

amazon-ion / ion-cli

Data missing when dumping gzip'd ion #126