search

intel / he-transformer

nGraph-HE: Deep learning with Homomorphic Encryption (HE) through Intel nGraph
Apache License 2.0
172 stars 35 forks source link

MNIST client-server example not working #39

Closed mlayou closed 4 years ago

mlayou commented 4 years ago

Hi,

I am testing the client-server MNIST example. What I'm seeing is one of two behaviours. Either both the client and server hung up without returning any answers, or one of them is killed. For more details, please see the logs below.

I have also tested the basic client-server example for matrix addition and multiplication (defined here), and it worked fine. Any thoughts on what's wrong with the MNIST client-server example? Is the system running out of RAM and killing the client process?

python test.py --backend=HE_SEAL \
               --model_file=models/cryptonets.pb \
               --enable_client=true \
               --encryption_parameters=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L8.json

(venv-tf-py3) user1@ubuntu:~/nGraph/he-transformer/examples/MNIST$ python test.py --backend=HE_SEAL --model_file=models/cryptonets.pb --enable_client=true --encryption_parameters=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L8.json
[...]
Model restored
loaded model
nodes ['import/input', 'import/convd1_1/kernel', 'import/convd1_1/bias', 'import/convd1_1/Conv2D', 'import/convd1_1/BiasAdd', 'import/activation/mul', 'import/Reshape/shape', 'import/Reshape', 'import/squash_fc_1/kernel', 'import/squash_fc_1/bias', 'import/squash_fc_1/MatMul', 'import/squash_fc_1/BiasAdd', 'import/activation_1/mul', 'import/output/kernel', 'import/output/bias', 'import/output/MatMul', 'import/output/BiasAdd']
2020-04-07 15:09:38.781680: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2496095000 Hz
2020-04-07 15:09:38.781958: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1693e40 executing computations on platform Host. Devices:
2020-04-07 15:09:38.782003: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
[WARN] 2020-04-07T19:09:38z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[WARN] 2020-04-07T19:09:38z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
2020-04-07 15:09:38.830897: I /home/user1/nGraph/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/ngraph_bridge/grappler/ngraph_optimizer.cc:239] NGraph using backend: HE_SEAL
2020-04-07 15:09:38.858285: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[WARN] 2020-04-07T19:09:38z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[WARN] 2020-04-07T19:09:38z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
2020-04-07 15:09:38.916244: I /home/user1/nGraph/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/ngraph_bridge/grappler/ngraph_optimizer.cc:239] NGraph using backend: HE_SEAL
[WARN] 2020-04-07T19:09:38z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[WARN] 2020-04-07T19:09:38z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
python pyclient_mnist.py --batch_size=1024 \
                         --encrypt_data_str=encrypt
(venv-tf-py3) user1@ubuntu:~/nGraph/he-transformer/examples/MNIST$ python pyclient_mnist.py --batch_size=1024 --encrypt_data_str=encrypt
[...]
[WARN] 2020-04-07T19:09:47z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
Killed

When I start the client before the server, it complains that the connection is refused. Once the server is started, that error is not displayed anymore. But the client either keeps waiting endlessly for the result from the server, or is killed.

[INFO] 2020-04-07T19:32:36z src/tcp/tcp_client.cpp 76   error connecting to server: Connection refused
[INFO] 2020-04-07T19:32:36z src/tcp/tcp_client.cpp 82   Trying to connect again
[INFO] 2020-04-07T19:32:36z src/tcp/tcp_client.cpp 76   error connecting to server: Connection refused
[INFO] 2020-04-07T19:32:38z src/tcp/tcp_client.cpp 82   Trying to connect again
mlayou commented 4 years ago

I've run the same example with a higher verbosity level. 

OMP_NUM_THREADS=1 \
NGRAPH_HE_VERBOSE_OPS=all \
NGRAPH_HE_LOG_LEVEL=3 \
python test.py \
  --backend=HE_SEAL \
  --model_file=models/cryptonets.pb \
  --enable_client=true \
  --encryption_parameters=$HE_TRANSFORMER/configs/he_seal_ckks_config_N11_L4_debug.json
OMP_NUM_THREADS=1 \
NGRAPH_HE_VERBOSE_OPS=all \
NGRAPH_HE_LOG_LEVEL=3 \
python pyclient_mnist.py \
  --batch_size=1024 
  --encrypt_data_str=encrypt

Here's the logs I 'm getting. Any recommendations on how to set the parameters so that the computations complete?

Server Side: 

(venv-tf-py3) user1@ubuntu:~/nGraph/he-transformer/examples/MNIST$ OMP_NUM_THREADS=1 NGRAPH_HE_VERBOSE_OPS=all NGRAPH_HE_LOG_LEVEL=3 python test.py --backend=HE_SEAL --model_file=models/cryptonets.pb --enable_client=true --encryption_parameters=/home/user1/nGraph/he-transformer/configs/he_seal_ckks_config_N11_L4_debug.json 
[...]
Model restored
loaded model
nodes ['import/input', 'import/convd1_1/kernel', 'import/convd1_1/bias', 'import/convd1_1/Conv2D', 'import/convd1_1/BiasAdd', 'import/activation/mul', 'import/Reshape/shape', 'import/Reshape', 'import/squash_fc_1/kernel', 'import/squash_fc_1/bias', 'import/squash_fc_1/MatMul', 'import/squash_fc_1/BiasAdd', 'import/activation_1/mul', 'import/output/kernel', 'import/output/bias', 'import/output/MatMul', 'import/output/BiasAdd']
2020-04-07 18:08:52.700559: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2496095000 Hz
2020-04-07 18:08:52.701235: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x25e2700 executing computations on platform Host. Devices:
2020-04-07 18:08:52.701286: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
[WARN] 2020-04-07T22:08:52z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[WARN] 2020-04-07T22:08:52z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_encryption_parameters.cpp 257  
/
| Encryption parameters :
|   scheme: CKKS
|   poly_modulus_degree: 1024
|   coeff_modulus size: 150 (30 + 30 + 30 + 30 + 30) bits
|   scale : 1.07374e+09
|   complex_packing: False
|   security_level: 0
\
2020-04-07 18:08:52.740725: I /home/user1/nGraph/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/ngraph_bridge/grappler/ngraph_optimizer.cc:239] NGraph using backend: HE_SEAL
2020-04-07 18:08:52.761366: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[WARN] 2020-04-07T22:08:52z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[WARN] 2020-04-07T22:08:52z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_encryption_parameters.cpp 257  
/
| Encryption parameters :
|   scheme: CKKS
|   poly_modulus_degree: 1024
|   coeff_modulus size: 150 (30 + 30 + 30 + 30 + 30) bits
|   scale : 1.07374e+09
|   complex_packing: False
|   security_level: 0
\
2020-04-07 18:08:52.795705: I /home/user1/nGraph/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/ngraph_bridge/grappler/ngraph_optimizer.cc:239] NGraph using backend: HE_SEAL
[WARN] 2020-04-07T22:08:52z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[WARN] 2020-04-07T22:08:52z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_encryption_parameters.cpp 257  
/
| Encryption parameters :
|   scheme: CKKS
|   poly_modulus_degree: 1024
|   coeff_modulus size: 150 (30 + 30 + 30 + 30 + 30) bits
|   scale : 1.07374e+09
|   complex_packing: False
|   security_level: 0
\
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 219    Compiling function with 1 parameters
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 223    Compiling function with parameter name Parameter_300 (Shape{})
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 102 Creating Executable
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 104 Parameter Parameter_300
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 108     shape Shape{} is not from client
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 133 Running optimization passes
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 185 Upadting HE op annotations
[INFO] 2020-04-07T22:08:52z src/pass/propagate_he_annotations.cpp 42    Running Propagate HE Annotations pass
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 161 ~HESealExecutable()
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 84 Setting config
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 90 Enabling client from config
[WARN] 2020-04-07T22:08:52z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[WARN] 2020-04-07T22:08:52z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_encryption_parameters.cpp 257  
/
| Encryption parameters :
|   scheme: CKKS
|   poly_modulus_degree: 2048
|   coeff_modulus size: 104 (30 + 22 + 22 + 30) bits
|   scale : 4.16973e+06
|   complex_packing: False
|   security_level: 0
\
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 111    Not masking garbled circuits inputs from config
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 118    Not masking garbled circuits outputs from config
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 104    Setting 1 garbled circuits threads from config
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 165    Tensor name: import/input with config HEOpAnnotation{from_client=True, encrypted=False, packed=True}
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 219    Compiling function with 1 parameters
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_backend.cpp 223    Compiling function with parameter name Parameter_560 (Shape{})
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 102 Creating Executable
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 104 Parameter Parameter_560
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 108     shape Shape{} is not from client
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 133 Running optimization passes
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 185 Upadting HE op annotations
[INFO] 2020-04-07T22:08:52z src/pass/propagate_he_annotations.cpp 42    Running Propagate HE Annotations pass
[INFO] 2020-04-07T22:08:52z src/seal/he_seal_executable.cpp 161 ~HESealExecutable()
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_backend.cpp 219    Compiling function with 1 parameters
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_backend.cpp 223    Compiling function with parameter name Parameter_820 (Shape{1, 28, 28, 1})
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_backend.cpp 227    Tag import/input
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 102 Creating Executable
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 104 Parameter Parameter_820
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 108     shape Shape{1, 28, 28, 1} is from client
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 113     Tag import/input
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 133 Running optimization passes
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 185 Upadting HE op annotations
[INFO] 2020-04-07T22:08:53z src/pass/propagate_he_annotations.cpp 42    Running Propagate HE Annotations pass
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 632 HESealExecutable::call
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 634 HESealExecutable::call validated inputs
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 255 Enable client
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 259 Starting server
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 301 Server accepting connections
[INFO] 2020-04-07T22:08:53z src/seal/he_seal_executable.cpp 281 Server waiting until session started
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_executable.cpp 309 Connection accepted
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_executable.cpp 313 Session started
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_executable.cpp 285 Server writing parameters message
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_executable.cpp 647 Waiting for m_client_inputs
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_executable.cpp 477 Server handling message
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_executable.cpp 388 Server setting inference tensor name import/input (corresponding to Parameter Parameter_820), with Shape{1, 28, 28, 1}
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_executable.cpp 396 Setting parameter Parameter_820 to packed
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_executable.cpp 403 Server sending inference of 1 parameters
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_executable.cpp 409 js {"function":"Parameter"}
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 477 Server handling message
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 531 Handling client tensors
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 608 Done loading client ciphertexts
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 652 Client inputs_received
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 656 Converting inputs to HETensor
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 665 Processing parameter Parameter_820(shape {Shape{1, 28, 28, 1}}) from client
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 722 Updating HE op annotations
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 185 Upadting HE op annotations
[INFO] 2020-04-07T22:09:00z src/pass/propagate_he_annotations.cpp 42    Running Propagate HE Annotations pass
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 725 Converting outputs to HETensor
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 732 Mapping function parameters to HETensor
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 746 Mapping function outputs to HETensor
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 773 [ Parameter_820 ]
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 787 Parameter shape Shape{1, 28, 28, 1} from client
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 773 [ Reshape_828 ]
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 826 encrypted_out 1
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 827 packed_out 1
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 1205    Shape{1, 28, 28, 1} reshape Shape{1, 1, 28, 28}
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 876 Reshape_828 took 0ms
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 773 [ Constant_1096 ]
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 777 Constant shape Shape{5, 1, 5, 5}
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 826 encrypted_out 0
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 827 packed_out 0
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 876 Constant_1096 took 0ms
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 773 [ Convolution_830 ]
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 826 encrypted_out 1
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 827 packed_out 1
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_executable.cpp 1043    Shape{1, 1, 28, 28} Conv Shape{5, 1, 5, 5} => Shape{1, 5, 14, 14}
[INFO] 2020-04-07T22:09:00z src/seal/kernel/convolution_seal.cpp 216    Finished out coord 0
[INFO] 2020-04-07T22:09:01z src/seal/kernel/rescale_seal.cpp 28 Rescaling 980 elements
[INFO] 2020-04-07T22:09:01z src/seal/kernel/rescale_seal.cpp 66 New chain index 1
[INFO] 2020-04-07T22:09:01z src/seal/kernel/rescale_seal.cpp 79 Rescale_xxx took 419ms
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 876 Convolution_830 took 1260ms
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 773 [ Reshape_831 ]
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 826 encrypted_out 1
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 827 packed_out 1
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 1205    Shape{1, 5, 14, 14} reshape Shape{1, 14, 14, 5}
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 876 Reshape_831 took 0ms
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 773 [ Constant_1097 ]
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 777 Constant shape Shape{1, 14, 14, 5}
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 826 encrypted_out 0
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 827 packed_out 0
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 876 Constant_1097 took 38ms
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 773 [ Add_833 ]
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 826 encrypted_out 1
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 827 packed_out 1
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 876 Add_833 took 44ms
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 773 [ Multiply_834 ]
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 826 encrypted_out 1
[INFO] 2020-04-07T22:09:01z src/seal/he_seal_executable.cpp 827 packed_out 1
[INFO] 2020-04-07T22:09:02z src/seal/kernel/rescale_seal.cpp 28 Rescaling 980 elements
[INFO] 2020-04-07T22:09:02z src/seal/kernel/rescale_seal.cpp 61 Skipping rescaling to chain index 0
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 876 Multiply_834 took 876ms
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 773 [ Reshape_835 ]
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 826 encrypted_out 1
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 827 packed_out 1
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 1205    Shape{1, 14, 14, 5} reshape Shape{1, 980}
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 876 Reshape_835 took 0ms
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 773 [ Constant_827 ]
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 777 Constant shape Shape{980, 100}
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 826 encrypted_out 0
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 827 packed_out 0
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 876 Constant_827 took 3ms
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 773 [ Dot_836 ]
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 819 Get output packing / encrypted
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 826 encrypted_out 1
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 827 packed_out 1
[INFO] 2020-04-07T22:09:02z src/seal/he_seal_executable.cpp 1075    Shape{1, 980} dot Shape{980, 100}
[ERR] 2020-04-07T22:09:02z src/seal/seal_util.cpp 215   new_scale 3.02295e+26 (87 bits) out of bounds
[ERR] 2020-04-07T22:09:02z src/seal/seal_util.cpp 217   Coeff mod bit count 52
terminate called after throwing an instance of 'ngraph::ngraph_error'
  what():  scale out of bounds
[2]+  Killed                  OMP_NUM_THREADS=1 NGRAPH_HE_VERBOSE_OPS=all NGRAPH_HE_LOG_LEVEL=3 python test.py --backend=HE_SEAL --model_file=models/cryptonets.pb --enable_client=true --encryption_parameters=/home/user1/nGraph/he-transformer/configs/he_seal_ckks_config_N13_L10_debug.json
Aborted (core dumped)
(venv-tf-py3) user1@ubuntu:~/nGraph/he-transformer/examples/MNIST$

Client Side:

(venv-tf-py3) user1@ubuntu:~/nGraph/he-transformer/examples/MNIST$ OMP_NUM_THREADS=1 NGRAPH_HE_VERBOSE_OPS=all NGRAPH_HE_LOG_LEVEL=3 python pyclient_mnist.py --batch_size=1024 --encrypt_data_str=encrypt
[...]
[WARN] 2020-04-07T22:08:57z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_client.cpp 51  Client input tensor: import/input
[INFO] 2020-04-07T22:08:57z src/tcp/tcp_client.cpp 65   Trying to connect TCP client
[INFO] 2020-04-07T22:08:57z src/tcp/tcp_client.cpp 73   Connected to server
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_client.cpp 408 Client handling message
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_client.cpp 125 Client handling encryption parameters message
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_client.cpp 134 Client loading encryption parameters from stream size 77
[WARN] 2020-04-07T22:08:57z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[WARN] 2020-04-07T22:08:57z src/seal/seal_util.cpp 39   Parameter selection does not enforce minimum security level
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_encryption_parameters.cpp 257  
/
| Encryption parameters :
|   scheme: CKKS
|   poly_modulus_degree: 2048
|   coeff_modulus size: 104 (30 + 22 + 22 + 30) bits
|   scale : 4.16973e+06
|   complex_packing: False
|   security_level: 0
\
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_client.cpp 100 Client sending public and relin keys
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_client.cpp 408 Client handling message
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_client.cpp 143 Client handling inference request
[INFO] 2020-04-07T22:08:57z src/seal/he_seal_client.cpp 207 Writing to tensor
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_client.cpp 210 Writing to pb tensors
[INFO] 2020-04-07T22:09:00z src/seal/he_seal_client.cpp 218 Client sending encrypted input with shape Shape{1024, 28, 28, 1}
[INFO] 2020-04-07T22:09:03z src/seal/he_seal_client.cpp 458 Client waiting for results
fboemer commented 4 years ago

The connectivity issues should be independent of the parameter selection. he_seal_ckks_config_N13_L8.json should be large enough for the Cryptonets example. The scale out of bounds is expected when using small parameter choices, such as he_seal_ckks_config_N11_L4_debug.json. IIRC, this example takes ~4GB RAM. You should be able to tell if your system is running out of RAM, e.g. by checking htop

mlayou commented 4 years ago

The problem is due to my system running out of memory. I have confirmed that with top. I have increased the size of my swap memory, and it's working fine now.

With the new HW configuration (6.5 GB of RAM, and 6 GB of swap memory), running the MNIST example with the he_seal_ckks_config_N11_L4_debug.json parameters, makes my system use 100% of the RAM and close to 70% of the swap memory.

Thanks for your help and feedback.

PereteanuGeorge commented 3 years ago

Hello,

I am running into the same problem. I was wondering how did you solve the issue with connectivity in the end since I am doing the same thing.

Thank you!