lightly-ai / lightly

A python library for self-supervised learning on images.
https://docs.lightly.ai/self-supervised-learning/
MIT License
3.16k stars 284 forks source link

Add FDATransform #1726

Open guarin opened 5 days ago

guarin commented 5 days ago

Add FDATransform based on Disentangling the Effects of Data Augmentation and Format Transform in Self-Supervised Learning of Image Representations.

The different subtransforms have already been added in https://github.com/lightly-ai/lightly/issues/1646 The goal now is to combine them all into a single FDATransform. Note that the image in the paper shows that the amplitude rescale and phase shift transforms are applied in parallel but we can implement them sequentially as the subtransforms handle the split into amplitude/phase components internally.

Tasks

Follow the BYOLTransform code on how to implement the different views and the full FDATransform: https://github.com/lightly-ai/lightly/blob/master/lightly/transforms/byol_transform.py

Each view transform must implement the following steps: Screenshot 2024-11-07 at 10 02 15

Parameters for view 1 and 2: Screenshot 2024-11-07 at 10 03 00

vectorvp commented 5 days ago

Hello @guarin, please assign this issue to me.

guarin commented 5 days ago

Thanks for looking into it!

vectorvp commented 4 hours ago

@guarin, while implementing this I see there is no probability parameters in AmplitudeRescaleTransform and PhaseShiftTransform. Should we add it or what is the workaround?

guarin commented 4 hours ago

Hi, you can use the RandomApply transform from torchvision