Run RP on EIS virtual cluster

[mturilli]

Now that we have gained access to the cloud, we need to test RP on their virtual cluster. It is a standard SLURM cluster with virtual nodes (when requested they are elastically deployed).

[AymenFJA]

Update regarding SMCE NASA: I interacted with "JupyterHub" machines. As Denis described in his email, these machines can be used to access an HPC cluster (AWS-based) called the Parallel cluster. From the "JupyterHub" machines, users can launch jobs (large scale).

Currently, I could not find any information regarding how to do that. So, I contacted Denis to see if he has documentation showing how to do that. Once we figure that out, it becomes a matter of instructing RCT (PSI-J/SAGA) to launch jobs on that cluster and bootstrap RP agent on the HPC Parallel cluster resources. My understanding is that the Parallel cluster has Slurm already installed on it.

[AymenFJA]

Update from Denis:

Hi Aymen,

There’s no documentation about the parallel cluster, unfortunately, because the EIS project was on a rapid development schedule at the beginning and we scrambled to put the system together and didn’t document much. Having said that, I could give you a short walkthrough for how this is done.

Before we do that, though, I’d like to get FACTS up and running on just the JupyterHub machines, without using the parallel cluster. The JupyterHub machines are not super powerful but I’d like to use those as a “proving ground” to make sure that small FACTS runs work there before we move to the parallel cluster. It would also help me to get a short demo from your team to see how FACTS/RADICAL work on the system.

[AymenFJA]

As a result we tested RCT-FACTS on the Jupyter machines and they passed. The notebook that we used is attached blow: https://gist.github.com/AymenFJA/9a89f990162c375120ab9cfdad01faa1

[AymenFJA]

Update: At this point EIS team and mainly Denis asked about a demo for RCT-FACTS in this demo the main idea is to run FACTS workflow (not dummy) and from the output generate some visualization about the workflow.

We discussed this with @AlexReedy and he is working on it to provide us with the demo.

[mturilli]

@AlexReedy ping

[AymenFJA]

Update: @AlexReedy and I worked on the new demo and we have a new notebook here: https://github.com/radical-collaboration/facts/blob/facts_eis/notebooks/EIS_Demo/RCT-FACTS.ipynb

[AlexReedy]

New demo successful runs the FAIR climate step and FittedISMIP/GrIS module as well as produces a time series plot of FittedISMIP/GrIS global sea level rise projections vs Global Mean Surface Air Temperature.

[AymenFJA]

This ticket will be closed as the purpose of this ticket was to run RCT on EIS cloud nodes, and this was done successfully. The next step is the demo progress, and it will be in a separate ticket here: https://github.com/radical-cybertools/radical.pilot/issues/3084