Open leinad87 opened 4 years ago
Hi @leinad87 - thank you for your question.
What you really want is parameterized kernels, but we're not "there" yet since that requires coordination across the entire jupyter stack. In lieu of parameterized kernels, you'd need to create different kernelspecs for each combination of parameters.
There are two ways EG integrates with YARN, via Spark and via Dask (python only).
For Spark, you'd configure your kernelspec (kernel.json) to include the appropriate Spark/Yarn parameters in the SPARK_OPTS env entry that get passed to spark-submit.
For Dask, you'd configure your kernelspec (kernel.json) to include the appropriate Dask/Yarn parameters - presumably in the DASK_OPTS env entry.
Thank you @kevin-bates. I tried DASK option for python deployments, however it doesn't can comunicate back to the client. I'm using jupyter hub + jupyter enterprise.
Temporal solution: using pyspark on yarn cluster mode with driver on cluster mode. Once the spark context is initialized, stop it to free executors resources.
Just to be clear, you're using jupyter hub + notebook server with the latter configured to hit a single enterprise gateway - is that correct?
Once the spark context is initialized, stop it to free executors resources.
I don't understand what this accomplishes. Could you please elaborate more on what you're trying to do and how your 'Temporal solution' is a solution at all?
I'm running jupyter hub for authentication purpose and single entry point for the users. Jupyter Hub uses Jupyter Gateway as spawner: c.Spawner.cmd = ['jupyterhub-singleuser', '--gateway-url=http://127.0.0.1:29128', '--GatewayClient.http_user={username}']
. Jupyter Gateway shall deploy ipynb files on top of YARN. The idea is that YARN administrate cluster resources for both, pyspark and python jobs.
Tricky part: python kernel is the pyspark environment, but once the spark context is initialized, I close it to free resources.
I see. So you're using spark/YARN to "distribute" the kernel into the cluster, but don't really need spark, so you close the spark context and continue running the kernel in YARN - interesting.
Yeah, it seems like you really want the launch using dask-yarn via the dask_python_yarn_remote
kernelspec.
Have you tried looking into the application logs to see what kinds of issues (if any) are being reported?
Sorry, I should have mentioned this previously. If you don't need a spark context, set your initialization-mode to none
.
Thank you @kevin-bates for your support! The problem with initalization mode in none is that application status in YARN stays in ACCEPTED and after a while is killed.
Oh yeah, that would be true. That slipped my mind. Sorry about that.
What you really want is the DASK kernelspec. Have you spent time troubleshooting that? It doesn't get a lot of use and I wonder if some recent changes broke something that is easily fixed.
Not yet, I'm stuck with impersonation and jupyter hub because native python kernel runs as root.
How shall I configure Jupyter EG to run a jupyter notebook on a yarn cluster? The idea is to be able to generate notebooks with limit resources, for example 5 cores and 8GB, and let YARN to select the best node in the cluster to run this notebook.