Feat/orthogonalization improvements

deel-ai / deel-lip

Build and train Lipschitz constrained networks: TensorFlow implementation of k-Lipschitz layers

https://deel-ai.github.io/deel-lip/

MIT License

89 stars 10 forks source link

Closed thib-s closed 1 year ago

thib-s commented 2 years ago

added following optimizations in normalizers.py:

in bjorck: choose between (WWt)W and W(WtW) to save time and memory on non-square matrices
added maxiter_bjorck and maxiter_spectral parameters
added globals SWAP_MEMORY and STOP_GRAD_SPECTRAL and function to set those (default values fallback to original behavior)

The way to set the two last parameters is subject to discussion, I chose this method to avoid clutter layers interfaces.

thib-s commented 1 year ago

updated by fixing following problems:

fixed mixxing param in layer.get_config
fixed maxiter to old defaults from 1.2
When u is None in power iteration it is initialized with tf.random.uniform instead of tf.ones to be consistent with initializers and layers.