search

ITISFoundation / osparc-simcore

🐼 osparc-simcore simulation framework
https://osparc.io
MIT License
43 stars 27 forks source link

Improve performance of AWS master job run time of simple job using osparc api #5734

Open wvangeit opened 4 months ago

wvangeit commented 4 months ago

I've identified a test case which I think we could use to improve the performance (speed) of the entire aws (master) system.

When running this script: https://github.com/wvangeit/osparc-pyapi-tests/blob/9d9e6b6703d5d6db56292be5ed949beceec3308a/noninter1/Makefile#L15 It will just run an extremely simple python runner, that reads a json file content, and dumps that content in a new json.

However, when running this 100 times, the average run time is ~5-10 minutes per job.

I think we should try to benchmark the different stage of this run, and try to get the runtime as low as possible. I would say in a first stage, ~1 min or so runtime would be more acceptable.

test.202404251047.txt

wvangeit commented 4 months ago

FYI @pcrespov @bisgaard-itis @sanderegg @matusdrobuliak66 I think this is something we should look at in a later stage. (I think i understand where most of this time is coming from, but we should see how we can improve it)