Speed up MultiQC

[pontushojer]

I did a profile of the current runtime using a full dataset, fully processed.

$ multiqc --profile-runtime .
[INFO   ]     multiqc_blr : Running MultiQC BLR Plugin v0.1.2.dev93+g73108f8
[INFO   ]     multiqc_blr : Running MultiQC BLR Plugin v0.1.2.dev93+g73108f8
[INFO   ]         multiqc : This is MultiQC v1.9
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Report title: BLR run report
[INFO   ]         multiqc : Searching   : /crex/proj/uppstore2018173/private/pontus/runs/200811.TELLSeq_test
Searching 1245 files..  [####################################]  100%          
...
[INFO   ]         multiqc : MultiQC complete
[INFO   ]         multiqc : Run took 119.83 seconds
[INFO   ]         multiqc :  - 113.41s: Searching files
[INFO   ]         multiqc :  - 2.92s: Running modules
[INFO   ]         multiqc :  - 0.15s: Compressing report data
[INFO   ]         multiqc : For more information, see the 'Run Time' section in multiqc_report_1.html

From this searching files takes the longest.

MultiQC also provides a plot of with searches takes up most of the time, see screenshot below.

Multiqc provides this section on runtime optimization: https://multiqc.info/docs/#optimising-run-time

[pontushojer]

Updates:

• https://github.com/NBISweden/BLR/commit/92d070ff139d827f8dbd07c01bfe169000df53e5: Limits MultiQC to only the used modules.
• https://github.com/NBISweden/BLR/commit/d453b4d1765b6e19cd433ffd227bf1548f2801cf: Skip .snakemake folder from search.

[pontushojer]

This is the current state of this issue. I tested the latest version of BLR on a resent dataset.

[INFO   ]         multiqc : Run took 3.32 seconds
[INFO   ]         multiqc :  - 0.80s: Searching files
[INFO   ]         multiqc :  - 1.73s: Running modules
[INFO   ]         multiqc :  - 0.20s: Compressing report data

From ~120 s to 3.3s is a huge improvement. With this I will close this issue.