search

replikation / poreCov

SARS-CoV-2 workflow for nanopore sequence data
https://case-group.github.io/
GNU General Public License v3.0
39 stars 16 forks source link

Number of CPUs #154

Closed karthick1087 closed 2 years ago

karthick1087 commented 2 years ago

Hi, sorry to bother with third consequtive query. I am running the pipeline in the AWS EC2 cloud instance. Even after I supplied with --ncores=16, some of the tools running with single cpu only, others are not engaged. Is there something that i can do to speed up the process.

hoelzer commented 2 years ago

Hi @karthick1087 firstly, I'm not super sure if we currently provide a ready-to-use AWS cloud profile. The pipeline was tested for GCP though. How did you start the pipeline, so what's you -profile parameter?

Besides, cloud compute resources are pre-configured here https://github.com/replikation/poreCov/blob/master/configs/nodes.config for each process. Passing options like --cores will not have an effect for cloud compute. You can also define your own configuration and inject this via -c my_config, for example when you want to adjust the cores and ram of certain processes.

replikation commented 2 years ago

hi @karthick1087 if you would like i could branch a AWS branch and we implement a generic AWS profile. i only have access to GCP and not AWS so i cant really test this wokflow.

karthick1087 commented 2 years ago

hi @karthick1087 if you would like i could branch a AWS branch and we implement a generic AWS profile. i only have access to GCP and not AWS so i cant really test this wokflow.

Hi thats great!!, If you Could branch for AWS. I will test the the work flow in AWS

replikation commented 2 years ago

@karthick1087 hi can you detail how you are currently starting was runs?

omarkr8 commented 2 years ago

curious about the cores now. Do I have to use --cores to get the pipeline to utilize them? seems like the initialisation parameters do report 0/X cores when i run porecov. how do --cores and --max_cores differ?

hoelzer commented 2 years ago

Hi!

--cores: Maximum cores used per process --max_cores: Maximum cores run at all

E.g. --cores 4 --max_cores 8 will run at most two 4core processes in parallel. We implemented this to prevent to many processes run in parallel on large work stations with many cores that are also used by others.

However, these params only effect local execution (-profile local). For cloud compute and on hpc (slurp, lsf,...) a pre-configured setup of cores per process is run.

On Tue, 12 Oct 2021, 07:37 omarkr8, @.***> wrote:

curious about the cores now. Do I have to use --cores to get the pipeline to utilize them? seems like the initialisation parameters do report 0/X cores when i run porecov. how do --cores and --max_cores differ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/replikation/poreCov/issues/154#issuecomment-940680341, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADN2CZ5AR7O4EXLUSCFOYYTUGPCQ5ANCNFSM5DSJG4ZQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.