libffcv / ffcv

FFCV: Fast Forward Computer Vision (and other ML workloads!)
https://ffcv.io
Apache License 2.0
2.8k stars 180 forks source link

The loading process is unusually slow #253

Open yajieC opened 1 year ago

yajieC commented 1 year ago

I have modified the two parameters as described in the documentation(https://docs.ffcv.io/parameter_tuning.html#scenario-large-scale-datasets). However, when I load data from “.beton” it takes a lot of time, so far it has been loaded for 6hours, but the loading process has not been completed.

Ps: The .beton file is about 320GB, with a total of about 130K images.