Eigendecomposition (ED) is widely used in deep networks. However, the backpropagation of its results tends to be numerically unstable, whether using ED directly or approximating it with the Power Iteration method, particularly when dealing with large matrices. While this can be mitigated by partitioning the data in small and arbitrary groups, doing so has no theoretical basis and makes its impossible to exploit the power of ED to the full. We introduce a numerically stable and differentiable approach to leveraging eigenvectors in deep networks. It can handle large symmetric square matrices without requiring to split them.
Pros:
Cons:
ZCA layer is nested at the bottom of the first residual block of ResNet. The 64 channels are partitioned into N groups and each group has d=64/N channels. The covariance matrix is computed for each group with the dimension of d.
Model | Dim. (d) | Max Acc. | Mean Acc. |
---|---|---|---|
ResNet18 | - | 93.02% | - |
ResNet18+ZCA | 4 | 95.41% | 95.29±0.11% |
ResNet18+ZCA | 8 | 95.57% | 95.38±0.18% |
ResNet18+ZCA | 16 | 95.60% | 95.37±0.14% |
ResNet18+ZCA | 32 | 95.54% | 95.36±0.15% |
ResNet18+ZCA | 64 | 95.56% | 95.41±0.09% |
the lr
is changed as follows:
0.1
for epoch [0,100)
0.01
for epoch [100,200)
0.001
for epoch [200,300)
0.0001
for epoch [300,350)
The pytorch must be GPU version, as we have not test the code on CPU machine. Now our code does not support multi-GPU setting. You need to run the following command to train the model. Here are the code for training ResNet18 on CIFAR10 & CIFAR100
On CIFAR10 Dataset: \
run ZCA whitening: python main.py --norm=zcanormsvdpi
\
run PCA denpoising: python main.py --norm=pcanormsvdpi
On CIFAR100 Dataset: \
run ZCA whitening: python main_cifar100.py --norm=zcanormsvdpi
\
run PCA denpoising: python main_cifar100.py --norm=pcanormsvdpi
The code might not be compatible with lower version of the specified packages.
Python = 3.7.2
PyTorch >= 1.1.0
Torchvision >= 0.2.2
Scipy >= 1.2.1
Numpy >= 1.16.3
tensorboardX
@inproceedings{wang2019backpropagation,
title={Backpropagation-Friendly Eigendecomposition},
author={Wang, Wei and Dang, Zheng and Hu, Yinlin and Fua, Pascal and Salzmann, Mathieu},
booktitle={Advances in neural information processing systems},
year={2019}
}
The code is heavily based on [pytorch-cifar][https://github.com/kuangliu/pytorch-cifar]