GuillaumeLeclerc
commented
2 years ago Hi @Abhishekvats1997!
This is a very good suggestion. However since we are a limited team I think it's better for the original authors to focus on the "core" of FFCV and add features that can't be implemented using the current API. This is unfortunately not the case for this feature. Everything exists in FFCV to add your own Fields and Decoders and we would be happy to assist you in your implementation. But I hope you understand we can't implement ourselves every Field that users might.
I'll leave the issue open if you have any question about custom Fields/Decoders or if someone else happen to want that particular Field and want to discuss it. If you happen to implement it make sure to submit a Pull request and we will be happy to review/merge it.
Hi, Thanks for the amazing work. Just wanted to point out a future enhancement i.e. when users have variable shaped tensors such as [8,2560,2560] and [8,2536,2536] as an example of their data samples and RandomCrops to generate tensors of the same shape so that their batch might be created, when one tries to implement such a use case in ffcv he is forced to take out the random transforms out of the dataset and shift to dataloader in order to maintain the randomness but is then left with variable shaped tensors which cannot be written to a file. Also, in cases of NLP tasks with variable length sequences some might like to have their own logic as part of the collate function (Pytorch Dataloader) to create their batch and want to write variable length sequences to disk again.