About Conv layer

[HamidFsian]

Hello, I read the tutorial 4. And I saw that you used only one conv2d. I want to apply 2 Conv layer. Is it possible using TenSEAL ?

Thank you

[MartinNoc]

Hi @HamidFsian, when using

ts.im2col_encoding(...)

for encryption and

y = enc_x.conv2d_im2col(kernel, windows_nb)

for the convolution layer like in Tutorial 4, indeed, only one Conv layer can be computed. This is because the special encoding/encryption method brings the input data into a certain format to allow for an efficient computation of the convolution. After the Conv layer (and packing) the data is stored in a CKKSVector and as TenSEAL does not support rotations, we cannot bring the data into the nice format as ts.im2col_encoding(...) does. Therefore, we cannot call conv2d_imcol(...) again.

To sum up:

• only one Conv layer possible (as mentioned)
• moreover, it has to be the first layer of the model
• client has to call the "special" encode/encrypt method, giving her/him information about the model (which is undesirable if we want to protect the model architecture from the client)

A solution to this can be to compute the convolution using a big matrix-vector-product. Using the kernel and all the other Conv attributes, one can compute a sparse matrix, that when multiplied with the input vector, gives the same result. The downside of this is, as we have to perform a lot more multiplications, this is around two orders of magnitude slower in execution time. To sum up again:

• input data is encrypted using the standard methods
• arbitrary position and number of Conv layers
• no model information is leaked
• BUT, big performance penalty

Watch out, as matrix-vector-multiplications are done in TenSEAL using the diagonal method, sparse matrices might exhibit a zero-diagonal resulting in a transparent ciphertext error. We already proposed a solution for this: https://github.com/OpenMined/TenSEAL/pull/405

Best, Martin

[HamidFsian]

Thank's for your explications.

Best regards, Hamid

[mayank64ce]

Hey @HamidFsian were you able to implement 2 convolutional layers ?

[MartinNoc]

We implemented arbitrary position and number of convolutional layers here: https://github.com/smile-ffg/he-man-tenseal/blob/ba732975f30ff6eb4309baf712200d6dea81885a/he_man_tenseal/inference.py#L382

As it might not be easy to comprehend, let me summarize:

• The model is in ONNX format
• We run the convolutional layer (represented by the GemmWrappedOperator) using a onnxruntime.InferenceSession using the parameters and attributes stored in the conv operator of the ONNX model.
• As input we use an identity matrix.
• The result of that run is the matrix of weights we use for the vector-matrix multiplication to perform the convolutional layer as described above in my previous comment.