libffcv / ffcv

FFCV: Fast Forward Computer Vision (and other ML workloads!)
https://ffcv.io
Apache License 2.0
2.8k stars 180 forks source link

Split Dataloader using indices #242

Open JJGO opened 1 year ago

JJGO commented 1 year ago

Trying to decide how to do 5-fold cross-validation with FFCV loading.

Does FFCV have any way of supporting loading just a subset of the indices, or should I have 5 files and loaders around for each split? Not sure how the overhead of multiple loaders stacks up (thinking about num_workers here).