replikation / What_the_Phage

WtP: Phage identification via nextflow and docker or singularity
https://mult1fractal.github.io/wtp-documentation/
GNU General Public License v3.0
103 stars 15 forks source link

deepvirfinder uses more cores unexpectly #189

Open Microbion opened 1 year ago

Microbion commented 1 year ago

--max_cores parameter can't limit deepvirfinder, because the theano backend to keras automatically uses all available cores for the internal matrix operations. as mentioned here https://github.com/jessieren/DeepVirFinder/issues/4 I wonder other tools have the similar issue, such as a numpy issue mentioned here https://stackoverflow.com/questions/30791550/limit-number-of-threads-in-numpy. Is it necessary and possible to add some environment variables for avoid using all cores?

mult1fractal commented 1 year ago

I think you have to ask the phage prediction tool developer directly as I cannot adjust the code of these tools

Are these commands you would use in a bash terminal before executing e.g. deepvirfinder? export MKL_NUM_THREADS=1 If yes I think its possible then

Microbion commented 1 year ago

I modified line 10 in workflows/process/deepvirfinder/deepvirfinder.wf, by OMP_NUM_THREADS=${task.cpus} dvf.py -c ${task.cpus} -i ${fasta} -o ${name}, and it worked. Consider of some tools' can't limit resource by itself or can but not correctly (not just deepvirfinder, I meeted similar troubles in other tools), I think it's necessary to limit resource manurally to improve robustness of workflow.

mult1fractal commented 1 year ago

nice! by chance do you know which tools have this issue?

I would update the code accordingly and create a new release candidate