search

deel-ai / deel-lip

Build and train Lipschitz constrained networks: TensorFlow implementation of k-Lipschitz layers
https://deel-ai.github.io/deel-lip/
MIT License
89 stars 10 forks source link

Orthogonal Initializer re-used #64

Closed Algue-Rythme closed 1 year ago

Algue-Rythme commented 1 year ago

Hi

I recently updated my local setting and I encountered the following warning:

/home/lbethune/anaconda3/envs/tf2/lib/python3.8/site-packages/keras/initializers/initializers_v2.py:120: UserWarning:

The initializer Orthogonal is unseeded and being called multiple times, which will return identical values  each time (even if the initializer is unseeded). Please update your code to provide a seed to the initializer, or avoid using the same initalizer instance more than once.

Versions:

I think it is because of the line: https://github.com/deel-ai/deel-lip/blob/bb0db0bdb2c39b219be385a8a09770f4d505381e/deel/lip/initializers.py#L24

The same object is kept, even after being processed by tensorflow.keras.initializers.get: https://github.com/keras-team/keras/blob/e6784e4302c7b8cd116b74a784f4b78d60e83c26/keras/initializers/__init__.py#L197

However, this is dangerous: the object instanciated in the __init__ is shared among all instances, as illustrated below:

class Dumbino():
  def __init__(self, l=[]):
    self.l = l
    pass
  def cool_fun(self, x):
    self.l.append(x)

dumb_1 = Dumbino()
dumb_1.cool_fun(42)
dumb_2 = Dumbino()
print(dumb_1.l, dumb_2.l)
# >> [42] [42]

This implies that identical matrices are built. This introduces strong bias in the model. The solution would be to re-instantiate the default initializer in every call to __init__.

thib-s commented 1 year ago

Thanks for your clear and reproducible issue. As you mentioned, this issue comes from the default parameter of the SpectralInitializer. A PR will be done soon, but if you want to avoid this problem, setting base_optimizer="orthogonal" in the SpectralInitializer or setting kernel_initializer="orthogonal" will fix this.