Open dskhudia opened 2 years ago
Hi! I think I misunderstood your question over slack---inserting ops into the pipeline after you make the loader isn't supported, as far as I know. @GuillaumeLeclerc can correct me if I'm wrong, but otherwise I'll close this issue since this isn't really on the roadmap---you can always just make two loaders off of the same .ffcv
file to achieve the same effect.
@andrewilyas Thanks a lot for the explanation. In https://github.com/mosaicml/composer, we have a set of data augmentation algorithms that can be applied flexibly anytime during the training. Dataloader by that time is already constructed so we have to modify the processing pipeline of the dataloader. Recreating (or having another dataloader) is not an option without losing current state.
Please consider adding this (modification of processing pipeline after dataloader construction ) as a feature. I think most of the machinery (For example, recompilation of pipeline) is already there in FFCV.
@dskhudia I see---one option is that you can make the transformations have a flag is_active, and if it's false just skip the transformation. Then during training you can do tx.is_active = True and tx.is_active = False. As long as you make the loader with the recompile=True option, this should do what you want.
Thanks. FFCV already respects the is_active flag?
Hi sorry for missing this! FFCV wouldn't see this is_active flag (you could call it anything), it would just be a parameter of the class, and then inside the function returned by generate_code
, you would just check is_active
and return the original image if is_active was False. Let me know if you need any further help here!
@andrewilyas Thanks. No problem. I see how this can work for the transformations I write. I was wondering how to make it work for the existing FFCV transformations.
@dskhudia I see, that makes sense---we'll consider working this into our next release, but for now my advice would be to accomplish what you want by subclassing the augmentation.
Thanks @andrewilyas , Eagerly waiting for the next release as it has some of the other improvments as well.
Adding a transformation in the pipeline once the loader object is constructed results in the following error.
Reproduction script: