Closed mlayou closed 5 years ago
fboemer
commented
5 years ago In general, setting OMP_NUM_THREADS=1 will produce a more meaningful error message for the ngraph::ngraph_error. I suspect the error might be a mismatch between the batch size on server and client (1024 vs. 2048).
fboemer
commented
5 years ago 
mlayou
commented
5 years ago Thanks! I'm currently testing the updated version. Does the new Makefile persist the build?
I'm asking because on the Makefile uses both the -v volume option as well as the --rm option in most of the targets (e.g., in the build_gcc target it usesdocker run --rm ${VOLUME}).
If --rm removes the container immediately after the docker build command completes, is it possible to run a container again mount it the previous volume without the need to rebuild from scratch?
The other shell scripts like make_docker_image.shand build-he-transformer-and-test.sh do also use the --rm option (e.g., docker build --rm=true). I guess the question above applies to them too.
fboemer
commented
5 years ago The new Makefile does not persist the build. Removing the --rm flag may persist the build, but we haven't tested this.
mlayou
commented
5 years ago Thanks for your help with this. I took the updated version, and run the example described at https://github.com/NervanaSystems/he-transformer/tree/master/examples
Following is what I get on the server side, and the client side, respectively.
Note how both of them are running on the same container. I made sure to run source $HE_TRANSFORMER/build/external/venv-tf-py3/bin/activate on the container. See below a list of the environment variables on the container.
Server logs
(venv-tf-py3) dockuser@9392d5fb239e:~/he-transformer-test/examples$ NGRAPH_ENABLE_CLIENT=1 NGRAPH_ENCRYPT_DATA=1 NGRAPH_TF_BACKEND=HE_SEAL python ax.py
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
2019-08-07 16:56:35.625511: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3498085000 Hz
2019-08-07 16:56:35.626573: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3c983b0 executing computations on platform Host. Devices:
2019-08-07 16:56:35.626598: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2019-08-07 16:56:35.629789: I /home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/src/ngraph_rewrite_pass.cc:247] NGraph using backend: HE_SEAL
[WARN] 2019-08-07T16:56:35z src/seal/he_seal_backend.cpp 59 Parameter selection does not enforce minimum security level
/
| Encryption parameters :
| scheme: CKKS
| poly_modulus_degree: 1024
| coeff_modulus size: 150 (30 + 30 + 30 + 30 + 30) bits
\
[INFO] 2019-08-07T16:56:35z src/seal/he_seal_backend.cpp 87 Scale 1.0737e+09
[INFO] 2019-08-07T16:56:35z src/seal/he_seal_executable.cpp 151 Setting up client in constructor
[INFO] 2019-08-07T16:56:35z src/seal/he_seal_executable.cpp 169 Enable client
[INFO] 2019-08-07T16:56:35z src/seal/he_seal_executable.cpp 173 Starting server
[INFO] 2019-08-07T16:56:35z src/seal/he_seal_executable.cpp 195 Server accepting connectionsClient logs
(venv-tf-py3) dockuser@9392d5fb239e:~/he-transformer-test/examples$ python pyclient.py
Traceback (most recent call last):
File "pyclient.py", line 17, in <module>
import he_seal_client
ImportError: No module named 'he_seal_client'
(venv-tf-py3) dockuser@9392d5fb239e:~/he-transformer-test/examples$ env
HOSTNAME=9392d5fb239e
TERM=xterm
OLDPWD=/home/dockuser/he-transformer-test
VIRTUAL_ENV=/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3
PATH=/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/home/dockuser/he-transformer-test/examples
PS1=(venv-tf-py3) ${debian_chroot:+($debian_chroot)}\u@\h:\w\$
RUN_UID=1004
SHLVL=1
HOME=/home/dockuser
_=/usr/bin/env
(venv-tf-py3) dockuser@9392d5fb239e:~/he-transformer-test/examples$ Currently the docker builds do not build the python wheel. You would need to install this (see here) before running the examples
mlayou
commented
5 years ago Thanks that solves it. The basic client-server example works now.
However, there seems to be another problem while trying the MNIST examples.
For example, here's the error I get when I run the Cryptonets example. There seems to be a problem with the config parser. I get the same error with Cryptonets-Relu as well.
(venv-tf-py3) dockuser@73d8c870f849:~/he-transformer-test/examples/MNIST/Cryptonets$ NGRAPH_ENCRYPT_DATA=1 \
> NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json \
> NGRAPH_TF_BACKEND=HE_SEAL \
> python test.py --batch_size=4096
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
2019-08-07 19:03:06.204579: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3498085000 Hz
2019-08-07 19:03:06.205193: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3aa5580 executing computations on platform Host. Devices:
2019-08-07 19:03:06.205211: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2019-08-07 19:03:06.214124: I /home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/src/ngraph_rewrite_pass.cc:247] NGraph using backend: HE_SEAL
2019-08-07 19:03:06.214564: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
2019-08-07 19:03:06.243597: I /home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/src/ngraph_rewrite_pass.cc:247] NGraph using backend: HE_SEAL
2019-08-07 19:03:06.340674: I /home/dockuser/he-transformer-test/BUILD-GCC/ext_ngraph_tf/src/ext_ngraph_tf/src/ngraph_rewrite_pass.cc:247] NGraph using backend: HE_SEAL
terminate called after throwing an instance of 'ngraph::ngraph_error'
what(): Error parsing NGRAPH_HE_SEAL_CONFIG: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
(venv-tf-py3) dockuser@73d8c870f849:~/he-transformer-test/examples/MNIST/Cryptonets$ This error probably means the .json file does not exist at the path indicated. Perhaps $HE_TRANSFORMER is not set?
mlayou
commented
5 years ago You're right. That's the problem. $HE_TRANSFORMER was not set.
Thanks Fabian! I really appreciate it.
Hi, I was trying to run the examples where a client is needed to carry out some of the non-polynomial computations (e.g., ReLu, etc.). The example that I'm referring to here specifically is the MNIST-MLP (but I've seen the same error with other examples such MNIST-Cryptonets-Relu).
As indicated in the instructions that come with example, I've run the following on the server side.
Then, I have connected to the same container as indicated below, and run the client side code on it. (
hopeful_mendeleevin the command line below is the name of the container where the server is running).Below are the server-side and client-side logs, respectively.
Server-side logs
Client-side logs
Any thoughts on the cause of this error?
Thanks.