Ignore file indexing for native data loader.

argonne-lcf / dlio_benchmark

An I/O benchmark for deep Learning applications

https://dlio-benchmark.readthedocs.io

Apache License 2.0

65 stars 30 forks source link

Ignore file indexing for native data loader. #215

Closed hariharan-devarajan closed 2 months ago

hariharan-devarajan commented 3 months ago

The sample building and native data loader case is needed only for DLIO created data loaders. For native data loaders which provide their own API;s they provide their own indexing and there this sampling can be ignored.