Refine structure/contents of the Jupyter-based interactive workflows section

[kathoef]

For the current draft material I would suggest at the moment:

ExaESM_WP4.ipynb

• [ ] focus the notebook on the larger picture aspects from this talk by @willirath
• [ ] boil down the exaESM project aspects to WP4 tasks: Jupyer-based interactive analysis on HPC, scheduling policies, Dask worker resilience, ... maybe it only makes sense to have the exaESM aspects at the end of the first part of the workshop, i.e. after the Dask demonstration examples were shown and attendees have a better idea of how Dask works and can be used?

introduction_to_the_workshop.ipynb

• [ ] focus on what Jupyter is, how to set it up and how to use it on the NEC Linux cluster frontends
• [ ] rename the notebook accordingly, e.g. jupyter_on_frontends.ipynb

Dask_LocalCluster_Example.ipynb

• [ ] choose more general name for the notebook, e.g. jupyter_compute_nodes_example.ipynb
• [ ] because in principal it would be possible to work with Jupyter on compute nodes and use a MATLAB, Ferret, R, ... Jupyter kernel; we should probably highlight the Jupyter kernel feature at this point? or already in the Jupyter on frontends notebook?
• [x] Python example: do a non-Dask Monte Carlo estimate of the number pi and demonstrate the scaling problem; then demonstrate how to solve this with a Dask LocalCluster (existing workflow with user friendly on-node resource management!)

Dask_Jobqueue_Example.ipynb

• [ ] demonstrate how to work with Dask jobqueue based on this existing material https://github.com/ExaESM-WP4/dask-jobqueue-configs/tree/v2020.02.20.1/nesh but by using the Monte Carlo pi estimate example?

add Summary.ipynb notebook

• [ ] provide pro/contra aspects from user perspective (we should hear about the system administrators perspective here and should get back to these points during the discussion section of the workshop after other potential solutions have been explored)
• [ ] refer back to exaESM research-oriented tasks? or put them here for the first time? (i.e. how to make compute node resources available for interactive analyses is a HPC system site-specific and by no means a generally solved/solvable problem)
• [ ] in this context present e.g. https://github.com/ExaESM-WP4/nesh-monitoring/blob/v2020.01.24.1/analysis.ipynb to give an overview on idle compute node resources on the NEC Linux cluster available for interactive analyses, e.g. both for Jupyter on compute node and/or Dask jobqueue worker approaches

[willirath]

I'm working on the LocalCluster example.

[willirath]

Dask_LocalCluster_Example.ipynb

• choose more general name for the notebook, e.g. jupyter_compute_nodes_example.ipynb because in principal it would be possible to work with Jupyter on compute nodes and use a MATLAB, Ferret, R, ... Jupyter kernel; we should probably highlight the Jupyter kernel feature at this point? or already in the Jupyter on frontends notebook?

Not sure about this. I think, I'd mention the kernel idea but not dive in too deep at this point.

(But we should prepare the setup of a Ferret kernel.)