kevin-bates
commented
4 years ago Hi @SevenMilk - thanks for opening the issue. There are a number of inconsistencies with what you state and what you provide so I suspect we'll need a few iterations here.
-
The logs show your kernelspec using
LocalProcessProxywhen the issue shows the correctYarnClusterProcessProxyin use:[D 2020-10-14 16:46:46.744 EnterpriseGatewayApp] Instantiating kernel 'Spark' with process proxy: enterprise_gateway.services.processproxies.processproxy.LocalProcessProxy
As a result, I'll need to see the logs relative to
YarnClusterProcessProxy. SinceLocalProcessProxyis not aRemoteProcessProxy, the response address template parameter ({response_address}) was left unfilled so the kernel launcher had no way to know where to send its connection information. I suspect the launcher crashed due to this missing parameter. -
You don't need any of the following env entries in your kernel.json file:
"EG_YARN_ENDPOINT": "http://172.24.0.12:8088/ws/v1/cluster", "EG_IMPERSONATION_ENABLED": "True", "EG_YARN_LOG_LEVEL" : "DEBUG", "EG_KERNEL_LAUNCH_TIMEOUT": "40", "KERNEL_LAUNCH_TIMEOUT" : "60", -
KERNEL_LAUNCH_TIMEOUTmust come from the client-side and is used by EG to determine when to give up waiting for a response from the remote kernel running in the YARN cluster. Because you're configuration wasn't right, we can hold off increasing it for now, but I suspect we'll probably need this increased since you need to load a large file with your kernel. -
I don't any evidence of automatic restarts happening. The initial start is timing out due to the configuration issues.
What is the reason for restarting the spark cluster kernel every time?
-
You should be using notebook with the
--gateway-urloption sincenb2kghas been in notebook since the 6.0 release. Use something like the following (where172.24.0.216:8888is the IP and port that EG is running on).jupyter notebook --gateway-url=http://172.24.0.216:8888None of the class mappings are necessary.
-
Although your kernel name is 'spark', you're referencing files from the out-of-the-box kernelspec examples (
kernels/spark_python_yarn_cluster/bin/run.sh). I would recommend leaving those alone and creating kernelspec hierarchies (bin and scripts sub-folders) for each kernelspec so that they can be individually tuned.
That's enough for now. Please send a new set of logs once these issues have been addressed. If you still have issues, you will need to look at the stdout/stderr logs via the YARN tools. These will contain the output produced by the launcher - which may have some issues with the environment of the node on which it lands since you can't install anaconda there.
SevenMilk
Hi, I want to use spark --deploy-mode=cluster on jupyter notebook, so I study EG and try to setup. My working environment is on anaconda python=3.5 version.
For some reasons, I cannot install anaconda to every node, but I found other solution, I use
--conf spark.yarn.dist.archives=/home/ericjiang/miniconda3/envs/spark_env.zip#sparkto zip the env upload to Yarn so the worker node has the same env.
After setting, run EG & notebook and import the spark cluster kernel. Next, jupyter notebook retry the spark cluster kernel repeatedly. This situation should not happened, and I have no idea what the is root cause of it.
I try to follow this web as following https://github.com/jupyter/enterprise_gateway/issues/600?fireglass_rsn=true to fix the problems. Unfortunately, it's still not working :(
Let me summarize my questions as below:
Attach my log, kernel.json file and conda list
RUN Command
Logs
kernel.json
conda list