libffcv / ffcv

FFCV: Fast Forward Computer Vision (and other ML workloads!)
https://ffcv.io
Apache License 2.0
2.84k stars 178 forks source link

Store the Augmented FFCV Data then Load it? #290

Closed realliyifei closed 1 year ago

realliyifei commented 1 year ago

Since some data augmentation techniques in transforms are not supported in FFCV (Ref. the end of #84)

Is it recommended to store the augmented data (e.g. crop, flip, color jitter, gray scale, gaussian blur) as beton first then load it directly? So we can augment them by transforms and at the same time avoid slowing the FFCV loading. But not sure whether there exists other costs that I was not aware of.

tavisshore commented 1 year ago

No because then you'll lose access to the original data - you'll be stuck with the performed transforms.

Have a beton of your original images and then either augment with FFCV techniques or post-loading

realliyifei commented 1 year ago

Thanks for the reply! Yes I agreed and did it later via zipping the FFCV data loader, and using both FFCV and torchvision's data augmentation (it is a pity that both this FFCV and meta's FFCV-SSL don't work well on the SSL's data augmentation)