search

eth-sri / eran

ETH Robustness Analyzer for Deep Neural Networks
Apache License 2.0
320 stars 103 forks source link

How to run custom network trained on another dataset? #84

Closed jkomyno closed 3 years ago

jkomyno commented 3 years ago

Hi, thank you for this project. It looks like tf-verify only expects 'mnist', 'cifar10', 'acasxu', 'fashion' as possible datasets. Assuming that my network is saved in ONNX format, which steps should I take to certify a network trained on another dataset?

jkomyno commented 3 years ago

Note: issue #23 may seem similar to mine, but it mentions a custom model running on mnist, which is already a supported dataset.

cherrywoods commented 3 years ago

I am not affiliated with this repository, but I have worked with it with networks for different datasets in recent time. Based on this experience I would recommend you just to read through tf_verify/__main__.py and see what needs to be changed. This should mostly be to adapt the code to a different number of input dimensions than ACAS Xu, MNIST and CIFAR10 have. The code outside __main__.py should not need to be changed, at least I didn't had to just to get the networks work.

If your dataset has 5 inputs like ACAS Xu, in most cases you should be able to use the ACAS Xu option without changing the code. You should provide different normalisation means and ranges via the command line. For other image classification networks besides MNIST and CIFAR10, I have not tried any others, but for me the same code worked for both, so I figure you should be able to use other networks as well.

jkomyno commented 3 years ago

Hi @cherrywoods, thank you for your answer. I suspected that the only way as of now was manually editing tf_verify/__main__.py. Thank you for your clarification.

cherrywoods commented 3 years ago

No problem. You can have a look at this file in my fork: https://github.com/cherrywoods/eran/blob/master/tf_verify/verification_interface.py. This code worked for me for all kinds of networks.