Check if Randomization of Parameters works on non-trivial torch.nn Modules

[dilyabareeva]

Problem: Currently we are only testing Conv and Linear layers in the Randomization metrics. For those, there is a reset_parameter() built in torch, which we could use instead. What about other torch.nn modules? Does our method work, for example, for transformer architectures?

Solution:

• include a super small transformer as a test suit -> run a randomization test.

[gumityolcu]

I find it safe to look at all parameters and randomize everything, which we do currently. If it is a trainable parameter, it will show up in model.parameters() assuming correct pytorch behaviour. So I don't think this is an issue. Right?

[dilyabareeva]

Sounds safe to me! I would just add a small test to check if this method also works for other common nn.modules

[dilyabareeva]

• It doesn't work
• That's why the current ModelRandomization version only randomizes Linear layers
• Possibly combine this issue with #164.