Support Jupyter Notebook

[cindytsai]

Support Jupyter Notebook / Lab UI

libyt now supports accessing interactive mode through Jupyter Notebook or Lab. Both serial (compile through g++) and parallel (compile through MPI) are supported. Can run python under MPI environment, gather output to root process, act like normal Python prompt, and has auto-completion feature.

Currently, this feature has only tested on local machines.

API

Use this feature through:

yt_run_JupyterKernel(const char* flag_file_name, bool use_connection_file)

libyt will stop and launch kernel if flag_file_name is found, and find empty port itself if use_connection_file is false.

If you want to specify connection info, you can refer to Connection File and the file name must be libyt_kernel_connection.json.

How to Use?

Compile libyt

1. Set -DINTERACTIVE_MODE=ON -DJUPYTER_KERNEL=ON

2. Currently only supports build from CMake (version >= 3.15). It is based on xeus-zmq, which needs other libraries when compiling :

   • xeus-zmq
   • xeus
   • ZeroMQ
   • cppzmq
   • OpenSSL
   • nlohmann_json

3. After installing libyt, set JUPYTER_PATH to <libyt-install-dir>/share/jupyter, so that jupyter knows how to find customized kernel. You can put it in .bashrc.

   export JUPYTER_PATH=<libyt-install-dir>/share/jupyter:$JUPYTER_PATH

Getting jupyter-client and jupyter_libyt

1. Get these:

   • jupyter_libyt
   • jupyter-client >= 8.0.0 tested (not sure if 7.x will work or not)
   • (Optional)jedi: for auto-completion. (If you have IPython, then this is probably already installed.)

2. Set LIBYT_KERNEL_INFO_DIR system environment variable when launching jupyter client.

   • jupyter_libyt (or more precisely, the libyt kernel provisioner) uses this path to find libyt kernel and its info.
   • Normally, it is the directory where your application executable is.

     export LIBYT_KERNEL_INFO_DIR=<your-application-root-dir>

3. Launch Jupyter Notebook or Lab:

   jupyter-lab # or jupyter notebook

Example

An example workflow of compiling libyt in parallel mode, and launch Jupyter Notebook or Lab:

1. Compile and install libyt:

   
   cd libyt # root directory of libyt

cmake -S . -B build-jupyter \ -G "Unix Makefiles" \ -DINTERACTIVE_MODE:BOOL=ON \ -DJUPYTER_KERNEL=ON \ -DMPI_PATH=/home/cindytsai/Softwares/openmpi/4.1.4-gnu \ -DPYTHON_PATH=/home/cindytsai/Softwares/python/python3.9 \ -Dxeus-zmq_DIR=/home/cindytsai/Packages/xeus-zmq-1.1.0/build \ -Dnlohmann_json_DIR=/home/cindytsai/Packages/json-3.11.2/build \ -Dxeus_DIR=/home/cindytsai/Packages/xeus-3.1.1/build \ -Dxtl_DIR=/home/cindytsai/Packages/xtl-master/build \ -Dcppzmq_DIR=/home/cindytsai/Packages/cppzmq-4.10.0/build

cmake --build build-jupyter cmake --install build-jupyter --prefix

2. Set `JUPYTER_PATH`:
```bash
export JUPYTER_PATH=<install-dir>/share/jupyter

3. Compile and link your application to libyt.
4. Run your application. (Remember to put flag file so that libyt will stop and launch the kernel)
5. Open another terminal and set LIBYT_KERNEL_INFO_DIR to where your application executable is. Then run jupyter-lab or jupyter notebook.

TODO before Merge

• [x] Doc
• [x] Test on remote.
• [ ] ~Make getting dependencies easier by getting and installing them using CMake.~
  • Will use other package manager, and put it in different PR.
• [ ] Check support of Makefile.
  • Actually, I want to drop the support for Makefile ... :crying_cat_face:
  • Will drop the support of Makefile in separate PR.
• [ ] Support other publishing format beside text
  • Put in different PR.
• [x] Support libyt defined command