Benchmark Alignment WF run time in dev

icgc-argo / workflow-roadmap

Roadmap and management for genomic data processing

GNU Affero General Public License v3.0

1 stars 0 forks source link

Benchmark Alignment WF run time in dev #43

Closed rosibaj closed 3 years ago

rosibaj commented 4 years ago

launch the wfs with AND without empty-dir usage in nextflow
vary the parameters on the song/score client cpu/parts -- when a lot of things are downloading we have a spike in nfs process (slower download, but more parallel jobs)
run in dev environment before promotion to QA
record the difference in times
information is found in relay

rosibaj commented 4 years ago

Removing until https://github.com/icgc-argo/roadmap/issues/585 done

rosibaj commented 3 years ago

@rtisma has an outstanding PR with issues resolved so testing CAN be done in sprint - @rosibaj check in on Monday for cleanup.

akachru-github commented 3 years ago

Benchmarking notes:

We will test in QA/DEV as we need to know whether emptyDir provides any sort of improvement before moving to production.
Find BAM with many lanes. Run in DEV with emptydir enabled (with NO parallel runs in QA). After this, run same BAM in DEV with Emptydir disabled (with NO parallel runs in QA)
Can do a 4x4 test using the same BAM.
Confirm RDPC QA/DEV cluster resources : RDPC dev/qa: 9 x c1.xlarge - 15 cores 128 GB RAM 
May need to shift resources from prod to QA/DEV is the comparison turns out to be very close, or only a moderate improvement.
Ensure scientific results are not altered when emptydir is enabled.
SSD's in prod are gravvy.

andricDu commented 3 years ago

We have reconfirmed again, via slack, that testing is to be done in the dev namespace, with no re-testing needed in QA namespace.

junjun-zhang commented 3 years ago

Here are the specific testing scenarios (confirmed via slack):

Scenario 1: enable emptyDir to simultaneously launch two runs: one for small BAM (b78b41c0-e7fe-486e-8b41-c0e7fe386e01), the other for the large BAM (2122397b-37cd-47ff-a239-7b37cd17ff4d), totally 2 runs;
Scenario 2: same as 1, but disable emptyDir;
Scenario 3: same as 1 but launch four runs for each input BAM, totally 8 runs;
Scenario 4: same as 3, but disable emptyDir.

The testing scenarios will be executed one by one with minimal (or none) other activities taking place on dev and qa. We will measure completion times and network traffic as indicators for performance improvements.

UPDATE: due to space constraint, not able to run the large BAM in scenario 3 and 4. Ran 8 parallel jobs on the small BAM instead.

junjun-zhang commented 3 years ago

Full results of the benchmarking is here: https://docs.google.com/spreadsheets/d/1c_0q5uAfeGXX8tjVWRkGKmKAEyomRaQNEedV_rmdTCQ/edit#gid=0

Here are some summary numbers: