search

jts / nanopore-paper-analysis

Code for nanopore paper
33 stars 5 forks source link

Parallel run #4

Open eta000 opened 8 years ago

eta000 commented 8 years ago

I am installing this pipeline on Stallo; a rocks distro cluster in Norway. I am struggling to understand how the number of parallel processes are defined. If I set the number of cores to 20 and the number of threads to 1 I would expect it to run 20 threads on one node, but I cannot see any difference from the 20core-4 thread or 16 core-4 thread cases in terms of number of poa processes running. This seems to choke the compute node entirely, or am I missing something?

jts commented 8 years ago

Sorry I missed this earlier. The number of POA processes that are run in parallel will be exactly the CORES argument. Note that the pipeline script is for a single node only - if you want to distribute it on a cluster you will have to do it manually or modify the parallel commands for your environment.

eta000 commented 8 years ago

Any reason for running multithreaded?

Fra: Jared Simpson notifications@github.com<mailto:notifications@github.com> Svar til: jts/nanopore-paper-analysis reply@reply.github.com<mailto:reply@reply.github.com> Dato: fredag 06. november 2015 23.27 Til: jts/nanopore-paper-analysis nanopore-paper-analysis@noreply.github.com<mailto:nanopore-paper-analysis@noreply.github.com> Kopi: Espen Tangen espen.tangen@uit.no<mailto:espen.tangen@uit.no> Emne: Re: [nanopore-paper-analysis] Parallel run (#4)

Sorry I missed this earlier. The number of POA processes that are run in parallel will be exactly the CORES argument. Note that the pipeline script is for a single node only - if you want to distribute it on a cluster you will have to do it manually or modify the parallel commands for your environment.

— Reply to this email directly or view it on GitHubhttps://github.com/jts/nanopore-paper-analysis/issues/4#issuecomment-154561781.

ParsaAkbari commented 8 years ago

I am slightly confused by this, because when running the make file we pass in the number of cores: make -f full-pipeline.make CORES=xx but in the make file itself 'CORES' variable is declared assigned value of 16. Presumably we do not need to change the declared variable within the file and only modify the parameter which we pass into the make file?

I also noticed that when first running the pipeline the daligner -t5 .... command appears four times on htop even though I set the number of cores = 2

jts commented 8 years ago

Hi Parsa,

The command line argument (CORES=xx) will override the default value in the Makefile.

Jared

On Tue, Jan 19, 2016 at 5:58 AM, Parsa Akbari notifications@github.com wrote:

I am slightly confused by this, because when running the make file we pass in the number of cores: make -f full-pipeline.make CORES=xx but in the make file itself 'CORES' variable is declared assigned value of

  1. Presumably we do not need to change the declared variable within the file and only modify the parameter which we pass into the make file?

— Reply to this email directly or view it on GitHub https://github.com/jts/nanopore-paper-analysis/issues/4#issuecomment-172817727 .