Sprint 2: 25-29 October 2021

[backeb]

Sprint 2: Progress towards using HPC

Objectives

• Work towards architecture options 3 and 4.

Sprint dates: 25-29 October 2021

Activities

• [x] @avgils get access to HPC (see https://github.com/c-scale-community/use-case-hisea/issues/12#issuecomment-939810102)
• [x] @avgils share IP addresses with support@hpc.grnet.gr cc @sebastian-luna-valero
• [ ] @avgils and @lorincmeszaros deploy Singularity container with Delft3D model
• [x] #14
• [x] @lorincmeszaros check internal performance problem see notes https://confluence.egi.eu/pages/viewpage.action?pageId=103161695

Outstanding actions from previous sprint (#12)

• [x] @nikosT review information provided https://confluence.egi.eu/pages/viewpage.action?pageId=103161695 and indicate what's missing
• [x] @avgils check if cron job works post service being up again. And if it works add second dataset to download. Move example scripts to Github repo.
• [ ] @sebastian-luna-valero and @kkoumantaros engage INFN for information about creating TOSCA templates
• [x] @lorincmeszaros add information about configuration (nodes vs cores, set up, how its run, infrastructure specs) to https://confluence.egi.eu/pages/viewpage.action?pageId=103161695 @nikosT to review
• [ ] Set up NFS server (network storage) and adjust data downloads to download to NFS server (@nikosT @yan0s @avgils)

In prep for Sprint 3

Kubernetes as a service

• [ ] @sebastian-luna-valero find support to help set up Kubernetes cluster on GRNET OpenStack
• [ ] @backeb at AMB raise issue of budget needed to maintaining k8s service (no budget allocated to GRNET for this)

[nikosT]

Concerning the first outstanding action, currently, no other info is missing apart from additional performance metrics that are described in activities.

UPDATE: @backeb I'm not sure though if accounting information (e.g. use case HPC Core*Hours etc.) are supposed to be added here or they are discussed in another WP.

[sebastian-luna-valero]

Hi,

Regarding my action points:

• TOSCA templates. Could you please share the "recipe for installing the CMEMS MOTU API & CDS API client and set up a cron job to automatically download the necessary data."? I think uploading instructions to this repository is the best option (code along some readme/docs).

• Regarding k8s on GRNET OpenStack. I understand that the goal of this second sprint (architecture options 3 and 4) is to move the heavy/all processing to the GRNET HPC since the Cloud has already been tested with VMs (architecture options 1 and 2). So why looking at k8s at this point? I am sorry I missed the last meetings but I was full booked with the EGI Conference, and therefore I might be missing something obvious.

Regarding @nikosT question about the accounting, @enolfc is working on that, and here you are the link: https://confluence.egi.eu/display/CSCALE/C-Scale+VA+installations

Best regards, Sebastian

[lorincmeszaros]

@sebastian-luna-valero : @avgils moved the recipee (scripts and guidelines) to the suggested repo. Can you now proceed with the TOSCA templates?

[sebastian-luna-valero]

Thanks @lorincmeszaros and @avgils

I am testing the code in https://github.com/c-scale-community/use-case-hisea/pull/15

When I run (disclaimer: not sure whether year 2010 is valid!):

python download_era5.py --year 2010 --output testfile

I get:

Exception: Missing/incomplete configuration file: $HOME/.cdsapirc

Could you please help?

Many thanks, Sebastian

[lorincmeszaros]

Hi Sebastian,

The year is not the problem, any year after 1979 is okay. As I see the CDS API key installation step is missing. We documented that on the confluence page but as I see it is missing from the github repo. I have added that. Here are the steps that you have to take:

Install the CDS API keyhttps://cds.climate.copernicus.eu/api-how-to#install-the-cds-api-key

• If you don't have an account, please self register at the CDS registrationhttps://cds.climate.copernicus.eu/user/register?destination=%2F%23!%2Fhome page and go to the steps below.
• If you are not logged, please loginhttps://cds.climate.copernicus.eu/user/login?destination=%2F%23!%2Fhome and go to the step below.
• Copy the code displayed beside (url and key), in the file $HOME/.cdsapirc (in your Unix/Linux environment).

After this the error message "Exception: Missing/incomplete configuration file: $HOME/.cdsapirc" should disappear.

[sebastian-luna-valero]

Thanks, @lorincmeszaros

Should we have a C-SCALE account for the CDS credentials, instead of using personal accounts?

[lorincmeszaros]

@sebastian-luna-valero: Good suggestion. I created a C-SCALE account and uploaded the .cdsapirc file in the hisea repo. You can use that.

[backeb]

Retro

Top: What worked well...

• Nothing to add...

Tip: What to improve

• Noticed the EODC MS Teams chat is not being used well
• Github conversations seems to be more useful

Sprint progress

• We have access to the HPC but have not been able to test the different architecture options

• To test the performance of the cloud-only workflow, we need a Kubernetes cluster so we can use the Argo workflow tool. Hence cloud-only workflow and performance not fully tested.

  • We can deploy a Kubernetes cluster right now
  • But maintaining a Kubernetes cluster operational is not straightforward
  • A Managed Kubernetes cluster as a Service is not yet available in C-SCALE

• [ ] @jkonarski @MZICloudferro does CloudFerro offer a Managed Kubernetes Cluster Service?

• Re automation and HPC

  • Keep in mind that you need to run the job in a queue, so you need to keep track of progress in the queue

• Re efficiency of the code

  • Bare metal has better speed, but efficiency is not good

• [ ] share Singularity image with @nikosT to test optimisation (@avgils)

• [x] explore if any agreements are required to share Singularity image with GRNET (@backeb)

• NFS server not ready yet

  • Being discussed at higher level
  • Will provide feedback in due course

• [ ] @nikosT and @yan0s to provide update on status of "Set up NFS server (network storage) and adjust data downloads to download to NFS server (@nikosT @yan0s @avgils)"

Objectives for next sprint

• [ ] Deploy Singularity container (@avgils)
• [ ] Compare container performance between different HPC set ups (e.g. using MPI library in Singularity container vs using MPI library installed on HPC). See #14 (@lorincmeszaros @avgils)
• [ ] Deploy Kubernetes cluster (@sebastian-luna-valero find person who could help GRNET during this sprint, @avgils / @lorincmeszaros provide compute and storage requirements)
• [ ] Re https://github.com/c-scale-community/use-case-hisea/issues/13#issuecomment-954703322 find a way to make this "secret". What are the implications of having this open? (@sebastian-luna-valero)

Sprint 3 dates

22-26 Nov, but confirm with @avgils because of C-SCALE LSDA sprint or 29/11-2/12

Project level activities

• @sebastian-luna-valero and @kkoumantaros engage INFN for information about creating TOSCA templates
• @sebastian-luna-valero find support to help set up Kubernetes cluster on GRNET OpenStack
• @backeb at AMB raise issue of budget needed to maintaining k8s service (no budget allocated to GRNET for this)

AOB

• How to redeploy Kubernetes cluster with scripts etc easily? This is something to think about.

[backeb]

@nikosT @yan0s regarding the action

• [ ] explore if any agreements are required to share Singularity image with GRNET (@backeb)

See below the feedback I have from Deltares, @avgils we can go ahead and share the Singularity image with GRNET.

Feedback

Hi Bjorn,

Yes, this is fine by me. The kernel release is in ~2 weeks. Anna knows where to find the image 😊 We will be happy to support them get things up and running because it gets us useful feedback.

@Anna van Gils I assume you are the technical point of contact for them? Feel free to get in touch with Adri when you have technical questions setting it up. If you/they get blocked, just set up a call with us (please include me as optional. If my agenda is limiting ignore it please).

As a system requirement: Can you ask them to ensure Intel MPI is installed on their cluster. We are using version 2021.2.0.216 ourselves but a version close to that should be fine.

Akhil Piplani Project Manager | Deltares

cc @lorincmeszaros @sebastian-luna-valero