search

argonne-lcf / dlio_benchmark

An I/O benchmark for deep Learning applications
https://dlio-benchmark.readthedocs.io
Apache License 2.0
70 stars 30 forks source link

Redundant shuffling #221

Open zhenghh04 opened 3 months ago

zhenghh04 commented 3 months ago

In reconfigure, global shuffling is performed. The data loader only have the indices of the local samples. It still has the shuffling on the pytorch / dali data loader which need to be fixed.

Please send another PR to fixed the redundant shuffling issue @hariharan-devarajan