Compressed TFRecord and indexes stored on s3

AlanRace commented 3 months ago

Version

1.38

Describe the bug.

Hello, I have been trying to use DALI to parse our large existing set of data stored as TFRecords (ZLIB compressed) on s3. I have encountered a few issues with this set-up:

As far as I can tell, there is currently no support in DALI for compressed TFRecords (likely due to the use and requirement of an index generated by tfrecord2idx)
When using uncompressed TFRecords stored on s3, it is not possible to store / read the index from s3 (e.g. as in minimum reproducible example)

Minimum reproducible example

from nvidia.dali.pipeline import Pipeline
import nvidia.dali.fn as fn
import nvidia.dali.tfrecord as tfrec
import nvidia.dali.types as types

batch_size = 10

pipe = Pipeline(batch_size=batch_size, num_threads=4, device_id=0)
with pipe:
    inputs = fn.readers.tfrecord(path="s3://path/to/data/data1.tfrecord",
                    index_path="s3://path/to/data/data1.idx", 
                    features={
                                'a': tfrec.FixedLenFeature((), tfrec.string, ""),
                                'b': tfrec.FixedLenFeature((), tfrec.string, ""),
                            }, random_shuffle=True)

    pipe.set_outputs(inputs['a'], inputs['b'])

pipe.build()

Relevant log output

No response

Other/Misc.

No response

Check for duplicates

[X] I have searched the open bugs/issues and have found no duplicates for this bug report

jantonguirao commented 3 months ago

Thank you for reporting this, @AlanRace. I can confirm it is indeed a bug, and we'll push a fix shortly. Regarding zlib compressed TFRecords, it is unfortunately not supported at the moment.

jantonguirao commented 3 months ago

https://github.com/NVIDIA/DALI/pull/5515 should fix this issue. It should be available through nightly builds a day or two after it gets merged.

NVIDIA / DALI