C-SCALE computing resources: take them or lose them!

[sebastian-luna-valero]

Hi!

This use case currently has the following resources allocated:

• Compute: 48 vCPUs at CreoDIAS, 20 vCPUs at GRNET Cloud plus the capacity at GRNET HPC
• Storage: 2 TB at CreoDIAS, 1 TB at GRNET Cloud plus 13 TB at GRNET HPC

Do you think that's enough for the next 9 months? C-SCALE currently has spare capacity that can be allocated now, so if you need to scale up, please let us know asap!

If we don't hear from you by Friday 30th Sept we will assume that you don't need more capacity and we will then reuse the spare capacity in C-SCALE for other use cases.

On the other hand, if you no longer need C-SCALE computing resources, please let us know as well as they will be reused properly.

Best regards, C-SCALE

[sebastian-luna-valero]

cc: @backeb @lorincmeszaros

[backeb]

Hi @sebastian-luna-valero

For the HiSea use case our next steps are:

• wrap the docker containers in a workflow engine such as snakemake
• figure out how to do a hybrid deployment to run parts of the workflow on the GRNET HPC, and other parts on the GRNET cloud.

We also want to do performance tests with the current cloud workflow, including:

• comparing performance on CPU-intensive VMs vs RAM-intensive VMs

We would run the workflow for longer periods, e.g. simulating one or multiple years instead of the 5 day simulations we've been testing on.

We would still like to use CREODIAS and GRNET for these tests, but I'm not sure if we will consume all allocated resources.

[enolfc]

Hi @backeb

Thanks for this input.

Some initial comments:

You have 1M CPU hours at the HPC, if we count from here to the end of the project (9 months, 730 hours in average per month), that's 152 cores running all time, which is not small amount.

Snakemake supports sending parts of the workflow to kubernetes and to a batch system, so I assume you could use the same tool for interacting both with cloud and HPC. Is that your plan? If so, could we get started by deploying kubernetes on the cloud allocation for GRNET?

[backeb]

Woah! Yeah that's a lot of HPC.

I've posted messages internally at Deltares to ask if anyone is interested in using HPC for their simulations.

I doubt we will consume all of that.

cc @lorincmeszaros

[backeb]

Hi @enolfc

I just spoke to a colleague about a potential HPC use case from Deltares, that requires A LOT of resources. They did a back of the envelope calculation on how much it would cost to run their experiments on a commercial provider and ended up with O(500,000EUR)... I'm pushing them to use our resources.

Challenge is that they would only really start working in Jan 2023, unless the can get some PMs internally. I don't have many PMs left in my C-SCALE budget, I need to save some for work next year. Will keep you posted and also ask them to provide some requirements.