search

OSC / bc_example_jupyter

Batch Connect - Example Jupyter Notebook Server
https://openondemand.org/
MIT License
2 stars 36 forks source link

JupyterLab on multiple nodes? #11

Open simonLeary42 opened 1 year ago

simonLeary42 commented 1 year ago

https://github.com/OSC/bc_example_jupyter/blob/94d29b4ba705cfe3b17df8ce78f584a4a2f986c5/manifest.yml#L7

https://github.com/OSC/bc_example_jupyter/blob/94d29b4ba705cfe3b17df8ce78f584a4a2f986c5/form.yml#L49

Would JupyterLab work as expected if run on multiple nodes? When the user logs in to their server, which node does the link lead to?

It was my understanding that the only way for processes to work together across multiple nodes is if they use message passing or memory parallelization. Is that not true? Does JupyterLab come with these features?

Looking around, I found that Princeton says that JupyterLab should be run on just one node: https://researchcomputing.princeton.edu/support/knowledge-base/jupyter

johrstrom commented 1 year ago

I think you're right that you'd need some fancy footwork in the script so that you only launch the notebook on one node.

We do something similar with Jupyter + Spark where we only start the Spark server on one node and we launch workers on the others.

That said - you have me questioning whether that works as expected and if we are indeed starting a notebook on both nodes.