search

reanahub / reana-demo-bsm-search

REANA example - BSM search
MIT License
6 stars 30 forks source link

check performance #21

Closed tiborsimko closed 4 years ago

tiborsimko commented 4 years ago

The reana-demo-bsm-search example, tagged 1.0.0, is finishing in:

$ reana-client status -w bsm
NAME   RUN_NUMBER   CREATED               STARTED               ENDED                 STATUS     PROGRESS
bsm    1            2020-09-07T09:20:51   2020-09-07T09:21:04   2020-09-07T09:30:44   finished   67/65
$ reana-client status -w bsm
NAME   RUN_NUMBER   CREATED               STARTED               ENDED                 STATUS     PROGRESS
bsm    3            2020-09-07T08:57:30   2020-09-07T08:57:31   2020-09-07T09:44:53   finished   67/65
$ reana-client status -w bsm
NAME   RUN_NUMBER   CREATED               STARTED               ENDED                 STATUS     PROGRESS
bsm    1            2020-09-07T09:58:56   2020-09-07T09:59:07   2020-09-07T10:06:19   finished   67/65
$ reana-client status -w bsm
NAME   RUN_NUMBER   CREATED               STARTED               ENDED                 STATUS    PROGRESS
bsm    1            2020-09-07T11:49:51   2020-09-07T11:50:40   2020-09-07T12:20:21   finished  67/65

In all cases, the BSM example to run was the same (the tagged 1.0.0 version), and all deployments were pre-loaded with necessary Docker images so that there is no docker image pulling time obstructing the picture. The only thing that changes is the hardware and the REANA version. And note that hardware does not play a role, since old machine with old REANA was faster than a new machine with new REANA.

It therefore seems that we have introduced a performance problem sometime in July-August this year. The difference is very visible for this demo workflow. It could be related to changes in r-w-e-yadage (py38? options?) or to changes in the REANA infrastructure itself (job monitoring? job controller? workflow execution?).

We should profile the BSM run in order to find the root cause and address the problem. We can start bisecting from O(60) days-old master version that is known to work fast.

mvidalgarcia commented 4 years ago

Some news on BSM slowness, all components in master and r-w-e-yadage in 9bf44e1. Finished in 9min.

$ cd reana-workflow-engine-yadage
$ git checkout 9bf44e1
$ reana-dev docker-build -b DEBUG=1 -c reana-workflow-engine-yadage --no-cache
$ reana-dev kind-load-docker-image -c reana-workflow-engine-yadage
$ reana-dev python-install-eggs
$ reana-dev cluster-undeploy
$ reana-dev cluster-deploy --mode debug --admin-email jdoe@example.org --admin-password 123456 --exclude-components r-ui
$ eval $(reana-dev client-setup-environment)
$ reana-client ping
$ cd ../reana-demo-bsm-search
$ reana-client run -w bsm
...
$ reana-client status -w bsm
NAME   RUN_NUMBER   CREATED               STARTED               ENDED                 STATUS     PROGRESS
bsm    1            2020-09-22T08:17:31   2020-09-22T08:17:32   2020-09-22T08:23:27   finished   67/65