search

mc2-project / secure-xgboost

Secure collaborative training and inference for XGBoost.
https://mc2-project.github.io/secure-xgboost/
Apache License 2.0
105 stars 32 forks source link

Distributed secure-xgboost tutorial example yields error #141

Closed ryanleh closed 3 years ago

ryanleh commented 3 years ago

Running the distributed Secure XGBoost tutorial produces the following errors:

ryan@ryan-dev-vm:~/secure-xgboost/demo/python/distributed$ ./run-local.sh 2
2021-04-08 00:12:11,735 INFO start listen on 10.0.9.8:9091
2021-04-08 00:12:11,736 DEBUG num of retry 0
2021-04-08 00:12:11,737 DEBUG num of retry 0
Creating enclave
Creating enclave
Remote attestation
Remote attestation
Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
2021-04-08 00:12:13,289 DEBUG Created SlaveEntry for node: 10.0.9.8
task NULL connected to the tracker
2021-04-08 00:12:13,297 DEBUG Created SlaveEntry for node: 10.0.9.8
task NULL connected to the tracker
task NULL got new rank 02021-04-08 00:12:13,299 DEBUG Recieve start signal from 10.0.9.8; assign rank 0task NULL got new rank 12021-04-08 00:12:13,465 DEBUG Recieve start signal from 10.0.9.8; assign rank 12021-04-08 00:12:13,465 INFO @tracker All of 2 nodes getting startedCreating training matrix from encrypted fileCreating training matrix from encrypted file
Traceback (most recent call last):
  File "distr-training.py", line 35, in <module>
    dtrain = xgb.DMatrix({username: HOME_DIR + "demo/data/agaricus.txt.train.enc"})
  File "/usr/local/lib/python3.7/site-packages/securexgboost-0.1-py3.7.egg/securexgboost/core.py", line 570, in __init__
    c_lengths))
  File "/usr/local/lib/python3.7/site-packages/securexgboost-0.1-py3.7.egg/securexgboost/core.py", line 203, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
securexgboost.core.XGBoostError: [16178407] /home/ryan/secure-xgboost/enclave/../include/enclave/crypto.h:248: verification failed -- mbedtls_pk_verify returned -17280Stack trace:  [bt] (0) _ZN4dmlc15LogMessageFatalD1Ev  [bt] (1) _ZL15verifySignature18mbedtls_pk_contextPhmS0_m  [bt] (2) _ZN14EnclaveContext22verifyClientSignaturesEPhmPPcPS0_Pm  [bt] (3) _Z18check_signed_inputRNSt3__119basic_ostringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEEPPcPPhPm  [bt] (4) XGDMatrixCreateFromEncryptedFile  [bt] (5) enclave_XGDMatrixCreateFromEncryptedFile
  [bt] (6) ecall_enclave_XGDMatrixCreateFromEncryptedFile
  [bt] (7) oe_handle_call_enclave_function
  [bt] (8) _handle_ecall

Traceback (most recent call last):
  File "distr-training.py", line 35, in <module>
    dtrain = xgb.DMatrix({username: HOME_DIR + "demo/data/agaricus.txt.train.enc"})  File "/usr/local/lib/python3.7/site-packages/securexgboost-0.1-py3.7.egg/securexgboost/core.py", line 570, in __init__    c_lengths))  File "/usr/local/lib/python3.7/site-packages/securexgboost-0.1-py3.7.egg/securexgboost/core.py", line 203, in _check_call    raise XGBoostError(py_str(_LIB.XGBGetLastError()))securexgboost.core.XGBoostError: [16178407] /home/ryan/secure-xgboost/enclave/src/data/data.cc:842: Encountered parser error:[16178407] /home/ryan/secure-xgboost/enclave/rabit/src/./ssl_socket.h:28:Stack trace:  [bt] (0) _ZN5rabit5utilsL9print_errEi  [bt] (1) _ZN5rabit6engine13AllreduceBase16TryAllreduceTreeEPvmmPFvPKvS2_iRKN3MPI8DatatypeEE  [bt] (2) _ZN5rabit6engine15AllreduceRobust11RecoverExecEPvmiiiPKc  [bt] (3) _ZN5rabit6engine15AllreduceRobust9AllreduceEPvmmPFvPKvS2_iRKN3MPI8DatatypeEEPFvS2_ES2_PKciSE_  [bt] (4) _ZN5rabit6engine10Allreduce_EPvmmPFvPKvS1_iRKN3MPI8DatatypeEENS0_3mpi8DataTypeENSA_6OpTypeEPFvS1_ES1_PKciSG_
  [bt] (5) _ZN7xgboost4data13SimpleDMatrixC1ENSt3__16vectorIPNS0_11FileAdapterENS2_9allocatorIS5_EEEEfi
  [bt] (6) _ZN7xgboost7DMatrix4LoadERNSt3__16vectorIKNS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEENS6_IS9_EEEEbbbPPcRS9_m
  [bt] (7) XGDMatrixCreateFromEncryptedFile
  [bt] (8) enclave_XGDMatrixCreateFromEncryptedFile

Stack trace:
  [bt] (0) _ZN4dmlc15LogMessageFatalD1Ev
  [bt] (1) _ZN7xgboost7DMatrix4LoadERNSt3__16vectorIKNS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEENS6_IS9_EEEEbbbPPcRS9_m
  [bt] (2) XGDMatrixCreateFromEncryptedFile  [bt] (3) enclave_XGDMatrixCreateFromEncryptedFile  [bt] (4) ecall_enclave_XGDMatrixCreateFromEncryptedFile  [bt] (5) oe_handle_call_enclave_function  [bt] (6) _handle_ecall  [bt] (7) oe_enter

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ryan/secure-xgboost/host/dmlc-core/tracker/dmlc_tracker/local.py", line 49, in exec_cmd
    raise RuntimeError('Get nonzero return code=%d on %s %s' % (ret, cmd, env))
RuntimeError: Get nonzero return code=1 on ['python3', 'distr-training.py'] {'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'SSH_CONNECTION': '73.252.231.20 60122 10.0.9.8 22', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', '_': '../../../host/dmlc-core/tracker/dmlc-submit', 'LANG': 'C.UTF-8', 'OLDPWD': '/home/ryan/secure-xgboost/demo/python/multiclient-cluster-remote-control/server', 'XDG_SESSION_ID': '1', 'USER': 'ryan', 'OPAQUE_DATA_DIR': '/home/ryan/opaque/data/', 'OE_SDK_PATH': '/opt/openenclave/', 'PWD': '/home/ryan/secure-xgboost/demo/python/distributed', 'HOME': '/home/ryan', 'CMAKE_PREFIX_PATH': ':/opt/openenclave/lib/openenclave/cmake:/opt/openenclave/lib/openenclave/cmake', 'SSH_CLIENT': '73.252.231.20 57585 22', 'TMUX': '/tmp/tmux-1000/default,2220,0', 'XDG_DATA_DIRS': '/usr/local/share:/usr/share:/var/lib/snapd/desktop', 'PRIVATE_KEY_PATH': '/home/ryan/opaque/private_key.pem', 'SPARK_SCALA_VERSION': '2.12', 'SPARK_HOME': '/opt/spark', 'OPAQUE_HOME': '/home/ryan/opaque', 'SSH_TTY': '/dev/pts/0', 'MAIL': '/var/mail/ryan', 'SHELL': '/bin/bash', 'TERM': 'screen-256color', 'TMUX_PANE': '%13', 'SHLVL': '3', 'LOGNAME': 'ryan', 'XDG_RUNTIME_DIR': '/run/user/1000', 'MODE': 'HARDWARE', 'PATH': '/home/ryan/.local/bin:/home/ryan/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/spark/bin:/opt/spark/sbin:/home/ryan/.local/bin:/opt/openenclave/bin:/home/ryan/.fzf/bin', 'PKG_CONFIG_PATH': ':/opt/openenclave/share/pkgconfig:/opt/openenclave/share/pkgconfig', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'DMLC_NUM_WORKER': '2', 'DMLC_NUM_SERVER': '0', 'DMLC_TRACKER_URI': '10.0.9.8', 'DMLC_TRACKER_PORT': '9091', 'DMLC_TASK_ID': '0', 'DMLC_ROLE': 'worker', 'DMLC_JOB_CLUSTER': 'local'}

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ryan/secure-xgboost/host/dmlc-core/tracker/dmlc_tracker/local.py", line 49, in exec_cmd
    raise RuntimeError('Get nonzero return code=%d on %s %s' % (ret, cmd, env))
RuntimeError: Get nonzero return code=1 on ['python3', 'distr-training.py'] {'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'SSH_CONNECTION': '73.252.231.20 60122 10.0.9.8 22', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', '_': '../../../host/dmlc-core/tracker/dmlc-submit', 'LANG': 'C.UTF-8', 'OLDPWD': '/home/ryan/secure-xgboost/demo/python/multiclient-cluster-remote-control/server', 'XDG_SESSION_ID': '1', 'USER': 'ryan', 'OPAQUE_DATA_DIR': '/home/ryan/opaque/data/', 'OE_SDK_PATH': '/opt/openenclave/', 'PWD': '/home/ryan/secure-xgboost/demo/python/distributed', 'HOME': '/home/ryan', 'CMAKE_PREFIX_PATH': ':/opt/openenclave/lib/openenclave/cmake:/opt/openenclave/lib/openenclave/cmake', 'SSH_CLIENT': '73.252.231.20 57585 22', 'TMUX': '/tmp/tmux-1000/default,2220,0', 'XDG_DATA_DIRS': '/usr/local/share:/usr/share:/var/lib/snapd/desktop', 'PRIVATE_KEY_PATH': '/home/ryan/opaque/private_key.pem', 'SPARK_SCALA_VERSION': '2.12', 'SPARK_HOME': '/opt/spark', 'OPAQUE_HOME': '/home/ryan/opaque', 'SSH_TTY': '/dev/pts/0', 'MAIL': '/var/mail/ryan', 'SHELL': '/bin/bash', 'TERM': 'screen-256color', 'TMUX_PANE': '%13', 'SHLVL': '3', 'LOGNAME': 'ryan', 'XDG_RUNTIME_DIR': '/run/user/1000', 'MODE': 'HARDWARE', 'PATH': '/home/ryan/.local/bin:/home/ryan/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/spark/bin:/opt/spark/sbin:/home/ryan/.local/bin:/opt/openenclave/bin:/home/ryan/.fzf/bin', 'PKG_CONFIG_PATH': ':/opt/openenclave/share/pkgconfig:/opt/openenclave/share/pkgconfig', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'DMLC_NUM_WORKER': '2', 'DMLC_NUM_SERVER': '0', 'DMLC_TRACKER_URI': '10.0.9.8', 'DMLC_TRACKER_PORT': '9091', 'DMLC_TASK_ID': '1', 'DMLC_ROLE': 'worker', 'DMLC_JOB_CLUSTER': 'local'}

and

ryan@ryan-dev-vm:~/secure-xgboost/demo/python/distributed$ ./run-distributed.sh 2
2021-04-08 04:46:05,116 INFO start listen on 10.0.9.8:9091
2021-04-08 04:46:05,117 DEBUG SSH-ing to 127.0.0.1:22
2021-04-08 04:46:05,118 DEBUG SSH-ing to 127.0.0.1:22
2021-04-08 04:46:06,541 DEBUG Created SlaveEntry for node: 10.0.9.8
task NULL connected to the tracker
2021-04-08 04:46:06,587 DEBUG Created SlaveEntry for node: 10.0.9.8
task NULL connected to the tracker
task NULL got new rank 0
2021-04-08 04:46:06,588 DEBUG Recieve start signal from 10.0.9.8; assign rank 0
task NULL got new rank 1
2021-04-08 04:46:06,767 DEBUG Recieve start signal from 10.0.9.8; assign rank 1
2021-04-08 04:46:06,767 INFO @tracker All of 2 nodes getting started
Creating enclave
Remote attestation
Creating training matrix from encrypted file
Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
Traceback (most recent call last):
  File "distr-training.py", line 35, in <module>
    dtrain = xgb.DMatrix({username: HOME_DIR + "demo/data/agaricus.txt.train.enc"})
  File "/usr/local/lib/python3.7/site-packages/securexgboost-0.1-py3.7.egg/securexgboost/core.py", line 570, in __init__
    c_lengths))
  File "/usr/local/lib/python3.7/site-packages/securexgboost-0.1-py3.7.egg/securexgboost/core.py", line 203, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
securexgboost.core.XGBoostError: [16178571] /home/ryan/secure-xgboost/enclave/../include/enclave/crypto.h:248: verification failed -- mbedtls_pk_verify returned -17280
Stack trace:
  [bt] (0) _ZN4dmlc15LogMessageFatalD1Ev
  [bt] (1) _ZL15verifySignature18mbedtls_pk_contextPhmS0_m
  [bt] (2) _ZN14EnclaveContext22verifyClientSignaturesEPhmPPcPS0_Pm
  [bt] (3) _Z18check_signed_inputRNSt3__119basic_ostringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEEPPcPPhPm
  [bt] (4) XGDMatrixCreateFromEncryptedFile
  [bt] (5) enclave_XGDMatrixCreateFromEncryptedFile
  [bt] (6) ecall_enclave_XGDMatrixCreateFromEncryptedFile
  [bt] (7) oe_handle_call_enclave_function
  [bt] (8) _handle_ecall

Creating enclave
Remote attestation
Creating training matrix from encrypted file
Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
Traceback (most recent call last):
  File "distr-training.py", line 35, in <module>
    dtrain = xgb.DMatrix({username: HOME_DIR + "demo/data/agaricus.txt.train.enc"})
  File "/usr/local/lib/python3.7/site-packages/securexgboost-0.1-py3.7.egg/securexgboost/core.py", line 570, in __init__
    c_lengths))
  File "/usr/local/lib/python3.7/site-packages/securexgboost-0.1-py3.7.egg/securexgboost/core.py", line 203, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
securexgboost.core.XGBoostError: [16178571] /home/ryan/secure-xgboost/enclave/src/data/data.cc:842: Encountered parser error:
[16178571] /home/ryan/secure-xgboost/enclave/rabit/src/./ssl_socket.h:28: 
Stack trace:
  [bt] (0) _ZN5rabit5utilsL9print_errEi
  [bt] (1) _ZN5rabit6engine13AllreduceBase16TryAllreduceTreeEPvmmPFvPKvS2_iRKN3MPI8DatatypeEE
  [bt] (2) _ZN5rabit6engine15AllreduceRobust11RecoverExecEPvmiiiPKc
  [bt] (3) _ZN5rabit6engine15AllreduceRobust9AllreduceEPvmmPFvPKvS2_iRKN3MPI8DatatypeEEPFvS2_ES2_PKciSE_
  [bt] (4) _ZN5rabit6engine10Allreduce_EPvmmPFvPKvS1_iRKN3MPI8DatatypeEENS0_3mpi8DataTypeENSA_6OpTypeEPFvS1_ES1_PKciSG_
  [bt] (5) _ZN7xgboost4data13SimpleDMatrixC1ENSt3__16vectorIPNS0_11FileAdapterENS2_9allocatorIS5_EEEEfi
  [bt] (6) _ZN7xgboost7DMatrix4LoadERNSt3__16vectorIKNS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEENS6_IS9_EEEEbbbPPcRS9_m
  [bt] (7) XGDMatrixCreateFromEncryptedFile
  [bt] (8) enclave_XGDMatrixCreateFromEncryptedFile

Stack trace:
  [bt] (0) _ZN4dmlc15LogMessageFatalD1Ev
  [bt] (1) _ZN7xgboost7DMatrix4LoadERNSt3__16vectorIKNS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEENS6_IS9_EEEEbbbPPcRS9_m
  [bt] (2) XGDMatrixCreateFromEncryptedFile
  [bt] (3) enclave_XGDMatrixCreateFromEncryptedFile
  [bt] (4) ecall_enclave_XGDMatrixCreateFromEncryptedFile
  [bt] (5) oe_handle_call_enclave_function
  [bt] (6) _handle_ecall
  [bt] (7) oe_enter

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ryan/secure-xgboost/host/dmlc-core/tracker/dmlc_tracker/ssh.py", line 62, in run
    subprocess.check_call(prog, shell = True)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'ssh -o StrictHostKeyChecking=no 127.0.0.1 -p 22 'export DMLC_NUM_WORKER=2; export DMLC_NUM_SERVER=0; export DMLC_TRACKER_URI=10.0.9.8; export DMLC_TRACKER_PORT=9091; export DMLC_ROLE=worker; export DMLC_NODE_HOST=127.0.0.1; cd /home/ryan/secure-xgboost/demo/python/distributed/; python3 distr-training.py'' returned non-zero exit status 1.

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ryan/secure-xgboost/host/dmlc-core/tracker/dmlc_tracker/ssh.py", line 62, in run
    subprocess.check_call(prog, shell = True)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'ssh -o StrictHostKeyChecking=no 127.0.0.1 -p 22 'export DMLC_NUM_WORKER=2; export DMLC_NUM_SERVER=0; export DMLC_TRACKER_URI=10.0.9.8; export DMLC_TRACKER_PORT=9091; export DMLC_ROLE=worker; export DMLC_NODE_HOST=127.0.0.1; cd /home/ryan/secure-xgboost/demo/python/distributed/; python3 distr-training.py'' returned non-zero exit status 1.
ryanleh commented 3 years ago

This appears to be a duplicate of https://github.com/mc2-project/secure-xgboost/issues/137