Reverse reversible transformations and apply transforms on gradients

[panpiort8]

closes #28 #29

To be done (rather in distinct issues):

• [x] Implement reversible robustness transformations (rotation, jittering etc.). I believe it would hugely help.
• [ ] Make all the experiments once again. I've performed only few again. Ihmo results are more interesting than before.

[mproszewska]

Change the name of PR and describe what it does.

[mproszewska]

Could you change PR title to "Reverse reversible transformations and apply transforms on gradients" or something like that?

[mproszewska]

Have you noticed any difference in how lr value impacts optimization? Previously, we were optimizing normalized images; now, gradient goes via normalization (parametrization), right? So values of gradient should differ (e.g. are scaled or something). So lr should probably be scaled too (e.g. previously best lr was 0.2, maybe now it's 0.8). Does it make sense?

[panpiort8]

I did not perform any hps search. But you're right: it's likely that now they are different.

[mproszewska]

am I right in saying that reversible functions are only able to reverse last transformation one time? because random parameters of the last call are stored and changed to None after reverse call? if so (or something similar), maybe add that info in docstring

[panpiort8]

am I right in saying that reversible functions are only able to reverse last transformation one time? because random parameters of the last call are stored and changed to None after reverse call? if so (or something similar), maybe add that info in docstring

yep, only the last transformation can only be reversed (and can be reversed only once). Changing the docs.

[mproszewska]

am I right in saying that reversible functions are only able to reverse last transformation one time? because random parameters of the last call are stored and changed to None after reverse call? if so (or something similar), maybe add that info in docstring

yep, only the last transformation can only be reversed (and can be reversed only once). Changing the docs.

I think that there are going to be issues when we try to implement diverse term in #34. We will need to be able to transform (and reverse) few images during one optimization step. Is there any walk around that with current code? (or any idea how to allow this). Worst case scenario, we won't allow certain transforms when using diversity term.

[panpiort8]

am I right in saying that reversible functions are only able to reverse last transformation one time? because random parameters of the last call are stored and changed to None after reverse call? if so (or something similar), maybe add that info in docstring

yep, only the last transformation can only be reversed (and can be reversed only once). Changing the docs.

I think that there are going to be issues when we try to implement diverse term in #34. We will need to be able to transform (and reverse) few images during one optimization step. Is there any walk around that with current code? (or any idea how to allow this). Worst case scenario, we won't allow certain transforms when using diversity term.

I'm not sure how the diverse term is going to look like, but I believe we can deal with it somehow. Cannot we just put batch of images to reversible transformation and then just reverse them? It will transform all the images in batch in the same way, but maybe it's not a problem. We can also make something like ReversibleTransformStack(batch_size, reversible_transform) that will store n (n=batch_size) copies of reversible_transform and apply them separately to every image in the batch. Anyway, I believe we can manage it somehow.

[panpiort8]

I believe PR is ready now.