AppTeam Std Simulations on S4L/AWS

drniiken commented 1 year ago

Background

As mentioned to some members of the Team, in today's 2PM Meeting Erdem and me faced the following situation and possible use-case:

Erdem's Team (in this particular case it's Cosimo) has to support standards activities (Mark Douglas) and for this purpose run a huge number of simulations - >10k simulations in total.
These are simulations of dipoles in combination with cSAR3Ds, in various combinations (angles, distances, positions, etc.).
EM-FDTD/iSolve simulations, must be run on GPU, sim size = ca. 150-200MCells, sim duration = a few hours.
All of it is totally scriptable (pre-, sim and post-pro); Cosimo is working on the "big script" right now.
Niels suggested (or rather demanded) - as an experiment - to run all of this on AWS, using S4L and the "cloud infrastructure" ... as a proof of concept/ feasibility test, and also in particular to get a feeling for cost for "such large studies".
Cost, in contrary to buying a new up-to-date HPC on-premise facility which we/Z43 would own (cost in the range of CHF 50k).
The AWS cost shall be different from the one attached to NIH/STRIDES, since this is pure Z43 in-house work for which we can't charge the NIH, obviously :-) (we can use, e.g., the AWS IT'IS account for which billing is on Niels' credit card)

ToDo

The task for the Team would thus be to setup the infrastructure which allows the AppTeam (Cosimo) to run the simulations on AWS: IMPORTANT: please use as much "existing infrastructure and technology" as possible. We do not want to interfere with the anyhow super sporty timeline for S4L web (full).

Deploy on "IT'IS AWS".
Could be S4L service on oSPARC or something like S4L lite but w/o the lite, i.e., larger simulations and GPU support.
We could, e.g., fix book 10 (or more) appropriate machines which run the tasks in parallel, script based (Cosimo), with GPU.
GUI access actually not needed, assuming all can be scripted.
Output files are not excessively big.

Thanks and best, Nik

### Sundae
- [ ] run part of the simulation @mguidon, @sanderegg

drniiken commented 1 year ago

Dear @eofli, please extend/correct, if needed. Thanks.

mrnicegyu11 commented 1 year ago

After an initial chat with cosimo, we established:

We need preliminary runtime benchmarks, at least on Cosimo's machine and a g4dn aws instance, before starting the big run
Worst-case time estimate as of today is 40min * 25 (points) * 24 (angles) * 28 (antennas) * 3 (distance)

After an initial chat with @sanderegg:

Sync with @mguidon / @colinRawlings once they are back is needed on: S4L-full-web state (any blockers?), Multi-GPU support?
computational gateway usage is likely not feasible in this sprint, using a second autoscaler or permanent personalized machines is likely the way to go.

After an initial chat with @Surfict :

Connecting two VPCs of different aws accounts is trivially possible, no special handling for the license server shall be required. Nota bene: Both VPCs need a seperate IP range that doesnt overlap

Additional considerations:

[ ] What happens in case of maintenances/release/(un)scheduled downtimes?
[ ] Prepare a cost-estimate after cosimo's initial benchmark

sanderegg commented 1 year ago

Goal for sprint Mithril

Use-case:

cluster connected to osparc.io, not paid by STRIDES (on ITIS account)
user starts jupyter-smash on osparc.io and generate inputs files
in jupyter-smash use the osparc-API to create jobs using isolve-gpu
these jobs should go to the separate dask-cluster
user logs out
user logs in later and gets progress
user get the results
get estimation of costs

Initial plan

we do it on staging-AWS (faster deployment of bugfixes/changes)

[x] connect a computer using separate ITIS account to osparc.io [@Surfict , @mrnicegyu11], test by creating 1 machine and run whatever service on it
[x] increase vcpu, number of instances limits on ITIS account [@Surfict , @mrnicegyu11]
[x] service specs of jupyter-smash to go to autoscaled machines, for initial testing with integrated isolve-gpu [@sanderegg]
[ ] initial testing of small simulation in jupyter-smash [@colinRawlings, CF]
[x] define what are the necessary resources for isolve-gpu [@colinRawlings, CF] -> might need to modify isolve-gpu or implement resource override
[x] dask-gateway update dependencies [@sanderegg]
[x] test connection still works [@sanderegg]
[x] deploy dask-gateway on separate cluster [@sanderegg]
[x] delete data in the jobs
[x] osparc-API selection of a different cluster [@pcrespov, @matusdrobuliak66]
[ ] widget to show how the progress goes in jupyter smash [@colinRawlings]
[x] POC demo with 2,3 machines
[ ] POC with 50 machines

sanderegg commented 1 year ago

Created a separate cluster with 2 g4dn.xlarge machines running in separate AWS account connected to staging.osparc.io. Deployed a osparc-dask-gateway and run many sleepers succeeded.

mguidon commented 1 year ago

Update on sprint Mithril

Done

Setup of workflow for cosimo (python scripts to create/analyze simulations)
Helper wrapper for submitting jobs to the cloud and getting progress information about running solvers
Define hardware specs for cluster
Cost/time estimate for solving
initial testing (POC) thanks to https://github.com/ITISFoundation/osparc-issues/issues/618:
- 200 jobs each with 1 sleepers:2.0.2 with resources overridden to use 0.1 CPU and 100MB RAM
- 22 EC2 instances of t2.large (8Gb, 2CPU) t2 instances which translates to max 22 comp. sidecars (dynamically adapted for now) each able to run up to 10 jobs depending on resources needed
- creation of the jobs took about 2.5 minute (~0.7job per second)
- running the jobs took about 3 minutes
- testing with 2000 sleepers and then 16000 sleepers are planned, also with longer running ones.

Todo

Cost/time estimate for post-processing
Publishing release 7.2 based services
Execute the whole thing

sanderegg commented 1 year ago

Goal for sprint Jelly Beans

Test & Fix osparc-dask-gateway until we can reliably run:
- external dask-gateway with >20 machines,
- run batches of 1000 jobs in parallel without issues,
Run multiple instances of iSolve on the same node
Cost/time estimate for post-processing
Publishing release 7.2 based services
Execute the whole thing

sanderegg commented 1 year ago

Update on sprint Jelly Beans

Done

Improved osparc-dask-gateway stability to successfully run 3x 2200 sleepers computational services in a row on an external separated cluster
Planning for S4l:Big use case

Ongoing

Improving oSparc platform to cope with 1000s of computational services running and streaming logs/progress etc
Preparing s4l to submit jobs into osparc

Open

Execute the whole thing

sanderegg commented 1 year ago

isolve 7.2 with VRAM
run subgroup of isolves
ideally the whole thing

mguidon commented 1 year ago

Update Watermelon

[ ] Paused for now

sanderegg commented 1 year ago

Notes:

Communicated that running on AWS would be too expensive? need clarifications
We still want to run some parts for testing, POCs

drniiken commented 1 year ago

I would still learn from feedback regarding the "spirit" of this use case:

large amount of simulations
scaling
cost
...

However, this must not be based on this specific case. Can be another use case (or example), giving more insights.

@mguidon

ITISFoundation / osparc-issues