backeb commented 2 years ago

Sprint 4: 10-14 Jan 2022

Sprint objectives and tasks

1. Test performance of "real" models on HPC

Using the MPI library in the Singularity container works.

[x] #24
- [x] @lorincmeszaros coordinate Deltares support with Akhil (prep action before or during next sprint)
[x] @avgils @lorincmeszaros add partitioning step to the container (at the moment partitioning has to be done on our side)

2. Progress on NFS server downloads

The nfs-server is setup with 300TB as a starting volume you can mount it using mount 192.168.0.85:/nfs-volume <path_to_mount> similarly you can add the following line in fstab to mount in on boot by adding the following line in the end 192.168.0.85:/nfs-volume <path_to_mount> nfs4 defaults 0 0

[x] #25

3. Learn about PaaS Orchestrator

[x] @backeb arrange meeting with @gdonvito for demo/training. Include @avgils @lorincmeszaros @sebastian-luna-valero
[ ] @gdonvito @maricaantonacci configure HiSea VO for PaaS Orchestrator (@sebastian-luna-valero to support)

Other outstanding tasks

See notes from previous sprint retro.

[x] @sebastian-luna-valero find secure place to store .cdsapirc file for the C-SCALE account on the Climate Data Store.
- [x] @lorincmeszaros to get in touch with ECMWF and ask how to do this for C-SCALE
[ ] @lorincmeszaros @kkoumantaros finalise GRNET signing the standard Delft3D FM Pre-release software licensing agreement (which includes both Docker and Singularity containers)
- [x] @lorincmeszaros send empty template to @kkoumantaros for GRNET legal to review

cc @yan0s @nikosT @kkoumantaros

backeb commented 2 years ago

@maricaantonacci I've added you to the HiSea use case team - welcome!

@maricaantonacci will help configure the PaaS Orchestrator to use for setting up a Kubernetes cluster on GRNET for the HiSea use case.

Notes from the meeting where the PaaS Orchestrator was demonstrated can be found here: https://confluence.egi.eu/display/CSCALE/2022-01-10++PaaS+Orchestrator+demo+for+C-SCALE+HiSea+use+case

maricaantonacci commented 2 years ago

Hi, I managed to federate GRNET-OPENSTACK with the PaaS Orchestrator. Now I need to test the deployment of a Kubernetes cluster. @kkoumantaros, @nikosT could you please increase the cores quota (it's saturated..20/20 cores are currently used)? Thanks in advance!

kkoumantaros commented 2 years ago

yup we will request an increase in the quota and get back to you asap

backeb commented 2 years ago

Retrospective

Tops

General happiness!

Tips

Netherlands: OPEN DAYCARE!!! :-)

Progress update

1. Test performance of "real" models on HPC

Did some testing with new Singularity container
Included partitioning in run script
Results added to confluence
One test still pending
Performance good, as expected
2 tests
- using MPI lib on HPC faster than using MPI lib in container
Performance better than Deltares bare-metal (h6 nodes)
using 80 cores / node could result in long time in que - setting correct number of cores and correct memory size would give you a more fair que time.
- [x] @avgils check sbatch parameters
[ ] rerun performance tests for longer period - can run up to 1 year

2. Progress on NFS server downloads

keep that for later to use in combination with PaaS Orchestrator and kubernetes cluster
[ ] activity for next sprint

3. PaaS Orchestrator

managed to federate GRNET-OPENSTACK with the PaaS Orchestrator.
need to test the deployment of a Kubernetes cluster.
[ ] @kkoumantaros, @nikosT increase the cores quota (it's saturated..20/20 cores are currently used)
[ ] test in prep of during next sprint

Other outstanding tasks

[ ] @lorincmeszaros @kkoumantaros finalise GRNET signing the standard Delft3D FM Pre-release software licensing agreement (which includes both Docker and Singularity containers)
what happens after agreement signed, what does GRNET want to do with the container?
- nothing :-) we have already benchmarked bare-metal vs container on Deltares/NL infra
- If we are unhappy with performance we could tackle this again
- For this activity is out of scope
operational workflow is the objective for the use case.

AOB

Could be useful to have a test que on the HPC

Objectives for next sprint

[ ] PaaS Orchestrator to set up Kubernetes cluster
[ ] Schedule jobs in Kubernetes cluster to download
[ ] Set up Argo workflow
[ ] 1-year performance test

Dates for the next sprint

31 Jan - 4 Feb

lorincmeszaros commented 2 years ago

@maricaantonacci How is the Kubernetes cluster testing going? Is it ready to use? If yes, @avgils could begin with setting up the (Argo) workflow on Thursday

maricaantonacci commented 2 years ago

@lorincmeszaros I'm waiting for GRNET people to increase the quota on Openstack. As soon as the quota is increased, I will be able to make some preliminary tests

lorincmeszaros commented 2 years ago

@kkoumantaros & @maricaantonacci : just checking the status of the quota increase and subsequent testing. @avgils is back and ready to work on the sprint objectives from our side. Please let us know if it is ready / can be done today / tomorrow. Otherwise @backeb we should schedule another sprint for the work when the Kubernetes cluster is ready to use. Thanks!

kkoumantaros commented 2 years ago

I'm afraid that the quota increase is not available yet, I will get back to you when its done.

backeb commented 2 years ago

@kkoumantaros what is the bottleneck? Is it related to the amount of VA allocated to us in this use case? What can we do to speed up the process?

backeb commented 2 years ago

Hi @kkoumantaros 👋

Do we have an update on the quota increase so that @maricaantonacci can test the PaaS Orchestrator and we can continue with the use case deployment?

This is currently blocking us from moving ahead with the use case deployment 🤔

backeb commented 1 year ago

tasks are deprecated

c-scale-community / workflow-coastal-hydrowaq