jupyter notebook environment

backeb commented 1 year ago

Hi @kkoumantaros and @LukaszKubowicz cc @sebastian-luna-valero @lorincmeszaros

For the final step of the workflow we were thinking it would be nice to be able to launch a jupyter notebook with some processing tools available.

I've started setting something up here: https://github.com/c-scale-community/use-case-hisea/tree/main/notebooks

This would be the approach that we use when running jupyter notebooks via docker on your local machine, but it wouldn't work when running on a remote VM, since I guess you don't expose ports from docker to the internet...

How could we launch a Jupyter Notebook environment so users can analyse the output from the model run?

sebastian-luna-valero commented 1 year ago

Hi,

@enolfc prepared this tutorial: https://docs.egi.eu/users/tutorials/jupyter-datahub-virtual-machine/

I have already added the VM image to both the eval.c-scale.eu and hisea.c-scale.eu VOs, which could be a good starting option for the HiSea workshop.

However, if you are already working on a VM, you can also achieve this. First, you need to configure the security groups to allow inbound connectivity to the ports where Jupyter Notebooks will run (e.g. 8888, and I can give specific help on this as well). Second, you can do something like:

# if you don't have conda already
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p conda-install
source conda-install/etc/profile.d/conda.sh 

# create the conda env
git clone https://github.com/c-scale-community/use-case-hisea.git
cd use-case-hisea/notebooks/
conda install mamba -c conda-forge --yes
mamba env create -f environment.yaml 
conda activate dfm_tools_env

# run jupyter lab
jupyter labextension install jupyter-matplotlib
jupyter-lab --port=8888 --no-browser --ip=0.0.0.0

In the http://127.0.0.1:8888/lab?token= output generated by the command above, you need to replace 127.0.0.1 with the public IP of your VM and paste that into your web browser.

Note that I have added two more dependencies to environment.yaml:

name: dfm_tools_env
channels:
  - conda-forge
dependencies:
  - pip
  - python=3.8
  - shapely>=1.7.0
  - cartopy 
  - pyepsg
  - geopandas
  - contextily 
  - xarray
  - dask
  - ipykernel
  - jupyterlab # added
  - nodejs # added
  - pip:
    - git+https://github.com/openearth/dfm_tools.git

I hope that helps!

Best regards, Sebastian

backeb commented 1 year ago

First, you need to configure the security groups to allow inbound connectivity to the ports where Jupyter Notebooks will run (e.g. 8888, and I can give specific help on this as well).

@sebastian-luna-valero could we jump on a call to figure this out?

backeb commented 1 year ago

This works now: https://github.com/c-scale-community/use-case-hisea/tree/main/notebooks#getting-the-above-to-run-on-an-openstack-virtual-machine-in-the-cloud

c-scale-community / workflow-coastal-hydrowaq

jupyter notebook environment #36