How to add support for some operations

[TafkR]

Hello, I'm very interesting by your project. However, I encounter issues with non support for some operations of a Neural Network architecture as a term by term multiplication or splits and concatenations.

I have studied a bit the implementation of the operations in ERAN and ELINA and the support of these operations does not seem possible to me with the ELINA's system of neurons and expressions.

Do you plan to add support for these types of operations in the future or any idea how I will be able to implement this? Thank you in advance for the help.

[GgnDpSngh]

Hi there,

Thanks for your interest in this project. You can treat the term by term multiplication as activation (e.g. ReLU) and add support for it at all the places in ERAN, i.e., adding multiplication wherever you see ReLU. One can compute a Zonotope approximation of the multiplication based on the following paper: https://arxiv.org/abs/1904.11803

DeepPoly approximation of the multiplication can be designed similarly. We did support concatenation before (https://github.com/eth-sri/eran/blob/d650e59d366f4ef40ac6fa94c200592fc53df16b/tf_verify/read_net_file.py#L103) but I will have to check if it still works with the latest changes. The support for split can be added by treating it as a new type of layer. We can add support for these features in the future if your analysis requires it.

Let me know if this answers your question.

Cheers, Gagandeep Singh

[TafkR]

Thank you for your answer.

I'm currently trying to understand how the activation functions are coded. Then I will take a look at this paper. I'll get back to you if I encounter some issues.

The split layer seems to be a good idea to implement as it is needed by a lot of neural network architecture.

Cheers

[TafkR]

Hi again,

I think I understand correctly how the activation functions are working but I still don't know how to use it to make a term by term multiplication. I was thinking about using residual layers to do it instead. What do you think of this idea ?

Thank you again in advance for your help

Cheers

[GgnDpSngh]

Hi there,

This depends on your architecture. If you have a feedforward network as in Fig. 3 (b) of https://openreview.net/pdf?id=HJxkvlBtwH then having the term multiplication as activation is more suitable, otherwise, if your network has branches that are joined using term multiplication then implementation as residual layers is preferable.

Cheers, Gagandeep Singh