search

MasaakiU / MultiplexNanopore

Other
10 stars 2 forks source link

Parallel processing with more than two CPUs #5

Closed jyelland closed 2 months ago

jyelland commented 2 months ago

I'm trying out version 0.2.17b2 because I was excited to see that parallel processing is now supported, hopefully making alignments much faster. I installed the latest version in a conda environment and I am calling savemoney.post_analysis on my local HPC cluster, with up to 36 nodes. However the alignment seems to be throttled at two CPUs:

n_cpu=2 (max=36)

Is this because of a setting in https://github.com/MasaakiU/MultiplexNanopore/commit/c9422e603c9114d3f146085297336b7079e0eb21 that limits n_cpu?

It's much faster than with one, but would be nice to take full advantage of parallelization.

MasaakiU commented 2 months ago

Hello Jamie, Thank you for using the latest version of savemoney! That's my bad, 0.2.17b2 is a beta version and I did not add any description about how to use it. If nothing is specified, savemney uses only 2 cpu by default. Please specify the number of cores by 

$ python -m savemoney.post_analysis path_to_sequence_directory save_directory -nc 36

if you use it via command line. If you use it in the python script, 

import savemoney
savemoney.post_analysis("path_to_sequence_directory", "save_directory", n_cpu=36)

should work. For your reference, 0.2.17b2 should be fine and safe to use as far as I checked, although it is a beta version.

jyelland commented 2 months ago

Nice, thanks @MasaakiU! I was hoping that I was just missing an input flag. I'm very excited to use the latest version of savemoney, which seems to have worked very nicely for demultiplexing plasmids containing unknown barcodes.

MasaakiU commented 2 months ago

My pleasure! Please feel free to send me any bug reports or requests etc. you may have in the future!