How to compare lipschitz and vanilla trained models?

[HeinrichAD]

Are certificates also available for non lipschitz trained models?

Example: If you look at your example 4 "HKR multiclass and fooling" and use the vanilla_export() already before the training (and train with CrossEntropyLoss), the certificate values looks much better. But the certificate guarantee $\epsilon \geq \mathcal{M}$ seems to be broken.

Examples values from example 4:

(modified version)

Vanilla

...
Epoch 9/10
loss: 0.6177 - acc: 0.7770 - KR: 5.8537 - val_loss: 0.6360 - val_acc: 0.7759 - val_KR: 5.9343
Epoch 10/10
loss: 0.6014 - acc: 0.7863 - KR: 6.0668 - val_loss: 0.6202 - val_acc: 0.7745 - val_KR: 6.1329

Image #     Certificate     Distance to adversarial
---------------------------------------------------
Image 0        1.638                1.49
Image 1        3.833                2.74
Image 2        1.566                1.76
Image 3        2.611                1.92
Image 5        0.134                0.13
Image 6        0.710                0.72
Image 7        1.772                1.15
Image 8        1.762                1.09
Image 9        0.265                0.29

Lipschitz

...
Epoch 9/10
loss: 9.6693 - acc: 0.7319 - KR: 0.7755 - val_loss: 9.9769 - val_acc: 0.7241 - val_KR: 0.7848
Epoch 10/10
loss: 9.5753 - acc: 0.7381 - KR: 0.8022 - val_loss: 9.6624 - val_acc: 0.7384 - val_KR: 0.8130

Image #     Certificate     Distance to adversarial
---------------------------------------------------
Image 0        0.285                1.85
Image 1        0.311                2.09
Image 2        0.076                0.63
Image 3        0.345                1.97
Image 5        0.055                0.48
Image 6        0.158                1.17
Image 7        0.216                1.19
Image 8        0.307                1.63
Image 9        0.037                0.31

Edit: Could torchlip.utils.evaluate_lip_const be should to compare lipschitz and vanilla trained models?

[thib-s]

Hi @HeinrichAD, Thanks for your interest in our project! The formula used to compute certificate require the function to be 1-lipschitz to be valid. The general formula should be : M=(top1 - top2)/(sqrt(2)*L) Where L is the Lipschitz constant of the network. ( see this paper and this paper )

The function torchlip.utils.evaluate_lip_const, is not advised for this use. It is important to recall that computing the Lipschitz constant of a network is a NP-hard problem. The function you mentioned use the naive method and is used for unit testing purposes. Using it in this context would probably give broken certificates.

Feel free to take a look at this paper learn more about methods to evaluate the Lipschitz constant of a network.

@cofri should we update the doc of this function ?

[HeinrichAD]

Thanks. I will look into the papers. I will close the issue and reopen it later if necessary. My intention was the compare Lipschitz and "vanilla" trained models.

As side note: If it's only for testing everything is fine, but torchlip.utils.evaluate_lip_const has no GPU support. Inside, torch.distributions.Uniform creates a new Tensor based on its low parameter. Currently, it's just a float value. Hence, it will result always in an CPU tensor. A possible solution would be to pass low as tensor which is on the same device like the inputs (x) or model.

[cofri]

Hello @HeinrichAD, To compare Lipschitz and "vanilla" models in terms of robustness, a standard way is to run L2 adversarial attacks. It is then possible to see which model is more robust to the attack. Note that this method does not give any guarantee; it's an empirical result.

On the other hand concerning robustness guarantees, the certificates can be computed using the formula given by @thib-s. This formulation is valid for both Lipschitz and "vanilla" models and involves the Lipschitz constant of the network. For a Lipschitz model, this is straightforward as we know the Lipschitz constant of all layers (default to 1 in deel-torchlip). However for a "vanilla" model, the Lipschitz constant computation is not trivial. But if you can compute it accurately, you can use the same formula to get certificates for the "vanilla" model.

Moreover, thank you for your tip on GPU support. It is indeed implemented for testing but we keep in mind your solution to extend to GPU support.