Getting 'HTTP 403: Failed to authenticate SSHClient with password-less SSH' in Enterprise Gateway logs for YARN Client mode but from where?

[sedatkestepe]

Hello, I wanted to evaluate Jupyter Enterprise Gateway on our edge node which has Hadoop (ecosystem) client binaries. The trouble I am living is as I open new Spark - Python (YARN Client Mode) notebook I receive 500 in client notebook logs from EG and receive 403 in EG logs but the source is kind of ambiguous in logging even though EG runs in debug mode.

Enterprise Gateway logs are below. What could be the source of 403? Thanks in advance

PS: I have an additional Spark installation (2.3.1 which I am planning to use after initial successful run) in PATH but it doesn't seems to be the problem since it is not included in appended CMD string.

[D 2018-07-23 19:40:11.403 EnterpriseGatewayApp] Found kernel spark_python_yarn_client in /opt/anaconda/anaconda2/share/jupyter/kernels
[D 2018-07-23 19:40:11.403 EnterpriseGatewayApp] Found kernel python2 in /opt/anaconda/anaconda2/share/jupyter/kernels
[I 180723 19:40:11 web:2106] 200 GET /api/kernelspecs (192.168.1.2) 2.92ms
[D 2018-07-23 19:40:11.576 EnterpriseGatewayApp] RemoteMappingKernelManager.start_kernel: spark_python_yarn_client
[D 2018-07-23 19:40:11.582 EnterpriseGatewayApp] Instantiating kernel 'Spark - Python (YARN Client Mode)' with process proxy: enterprise_gateway.services.processproxies.distributed.DistributedProcessProxy
[D 2018-07-23 19:40:11.583 EnterpriseGatewayApp] Response socket launched on 192.168.1.3, port: 46938 using 5.0s timeout
[D 2018-07-23 19:40:11.583 EnterpriseGatewayApp] Starting kernel: [u'/usr/local/share/jupyter/kernels/spark_python_yarn_client/bin/run.sh', u'/run/user/0/jupyter/kernel-21b04bd9-12ea-4749-801c-3ca20ac2f0bc.json', u'--RemoteProcessProxy.response-address', u'192.168.1.3:46938', u'--RemoteProcessProxy.port-range', u'0..0', u'--RemoteProcessProxy.spark-context-initialization-mode', u'lazy', u'--EnterpriseGatewayApp.authorized_users', u"['guest','sedat']"]
[D 2018-07-23 19:40:11.584 EnterpriseGatewayApp] Launching kernel: Spark - Python (YARN Client Mode) with command: [u'/usr/local/share/jupyter/kernels/spark_python_yarn_client/bin/run.sh', u'/run/user/0/jupyter/kernel-21b04bd9-12ea-4749-801c-3ca20ac2f0bc.json', u'--RemoteProcessProxy.response-address', u'192.168.1.3:46938', u'--RemoteProcessProxy.port-range', u'0..0', u'--RemoteProcessProxy.spark-context-initialization-mode', u'lazy', u'--EnterpriseGatewayApp.authorized_users', u"['guest']"]
[D 2018-07-23 19:40:11.585 EnterpriseGatewayApp] BaseProcessProxy.launch_process() env: {'EG_MAX_PORT_RANGE_RETRIES': '5', u'SPARK_OPTS': u'--master yarn --deploy-mode client --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID}', u'SPARK_YARN_USER_ENV': u'PYTHONUSERBASE=/home/yarn/.local,PYTHONPATH=${HOME}/.local/lib/python2.7/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip,PATH=/opt/anaconda2/bin:$PATH', 'KERNEL_ID': u'21b04bd9-12ea-4749-801c-3ca20ac2f0bc', u'PYTHONPATH': u'${HOME}/.local/lib/python2.7/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip', 'KERNEL_GATEWAY': '1', 'EG_MIN_PORT_RANGE_SIZE': '1000', 'EG_IMPERSONATION_ENABLED': 'False', u'KERNEL_USERNAME': u'guest', u'SPARK_HOME': u'/usr/hdp/current/spark2-client', 'PATH': '/opt/anaconda/anaconda2/bin:/opt/anaconda/anaconda2/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/spark/spark-2.3.1-bin-hadoop2.7/bin', u'PYSPARK_PYTHON': u'/opt/anaconda2/bin/python', u'LAUNCH_OPTS': u'', u'KERNEL_PASSWORD': u'guest-password'}
[D 2018-07-23 19:40:11.585 EnterpriseGatewayApp] Invoking cmd: 'export KERNEL_USERNAME="guest";export KERNEL_ID="21b04bd9-12ea-4749-801c-3ca20ac2f0bc";export EG_IMPERSONATION_ENABLED="False";export SPARK_OPTS="--master yarn --deploy-mode client --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID}";export KERNEL_USERNAME="guest";export PYTHONPATH="${HOME}/.local/lib/python2.7/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip";export SPARK_YARN_USER_ENV="PYTHONUSERBASE=/home/yarn/.local,PYTHONPATH=${HOME}/.local/lib/python2.7/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip,PATH=/opt/anaconda2/bin:$PATH";export SPARK_HOME="/usr/hdp/current/spark2-client";export PYSPARK_PYTHON="/opt/anaconda2/bin/python";export LAUNCH_OPTS="";export KERNEL_PASSWORD="guest-password";nohup /usr/local/share/jupyter/kernels/spark_python_yarn_client/bin/run.sh /run/user/0/jupyter/kernel-21b04bd9-12ea-4749-801c-3ca20ac2f0bc.json --RemoteProcessProxy.response-address 192.168.1.3:46938 --RemoteProcessProxy.port-range 0..0 --RemoteProcessProxy.spark-context-initialization-mode lazy --EnterpriseGatewayApp.authorized_users ['guest'] >> /tmp/kernel-21b04bd9-12ea-4749-801c-3ca20ac2f0bc.log 2>&1 & echo $!' on host: localhost
[E 2018-07-23 19:40:11.605 EnterpriseGatewayApp] Failed to authenticate SSHClient with password-less SSH
[E 2018-07-23 19:40:11.605 EnterpriseGatewayApp] Failure occurred starting remote kernel on '127.0.0.1'. Exception: 'HTTP 403: Failed to authenticate SSHClient with password-less SSH'.
[E 180723 19:40:11 web:2106] 500 POST /api/kernels (192.168.1.2) 32.33ms

[kevin-bates]

@sedatkestepe - thank you for checking out Enterprise Gateway. Would you mind creating this issue in the EG repo? This really isn't an NB2KG issue.

The 403 is due to the SSH configuration within the EG/Yarn cluster.

[sedatkestepe]

Sorry. Really sorry.

[kevin-bates]

No worries - thanks for moving this to the EG repo. Closing here.