FrickTobias / BLR

MIT License
6 stars 5 forks source link

Speed up MultiQC #224

Closed pontushojer closed 4 years ago

pontushojer commented 4 years ago

I did a profile of the current runtime using a full dataset, fully processed.

$ multiqc --profile-runtime .
[INFO   ]     multiqc_blr : Running MultiQC BLR Plugin v0.1.2.dev93+g73108f8
[INFO   ]     multiqc_blr : Running MultiQC BLR Plugin v0.1.2.dev93+g73108f8
[INFO   ]         multiqc : This is MultiQC v1.9
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Report title: BLR run report
[INFO   ]         multiqc : Searching   : /crex/proj/uppstore2018173/private/pontus/runs/200811.TELLSeq_test
Searching 1245 files..  [####################################]  100%          
...
[INFO   ]         multiqc : MultiQC complete
[INFO   ]         multiqc : Run took 119.83 seconds
[INFO   ]         multiqc :  - 113.41s: Searching files
[INFO   ]         multiqc :  - 2.92s: Running modules
[INFO   ]         multiqc :  - 0.15s: Compressing report data
[INFO   ]         multiqc : For more information, see the 'Run Time' section in multiqc_report_1.html

From this searching files takes the longest.

MultiQC also provides a plot of with searches takes up most of the time, see screenshot below.

Screenshot 2020-08-13 at 10 30 46

Multiqc provides this section on runtime optimization: https://multiqc.info/docs/#optimising-run-time

pontushojer commented 4 years ago

Updates:

pontushojer commented 4 years ago

This is the current state of this issue. I tested the latest version of BLR on a resent dataset.

[INFO   ]         multiqc : Run took 3.32 seconds
[INFO   ]         multiqc :  - 0.80s: Searching files
[INFO   ]         multiqc :  - 1.73s: Running modules
[INFO   ]         multiqc :  - 0.20s: Compressing report data

From ~120 s to 3.3s is a huge improvement. With this I will close this issue.