Closed karthick1087 closed 2 years ago
Hi @karthick1087 firstly, I'm not super sure if we currently provide a ready-to-use AWS cloud profile. The pipeline was tested for GCP though. How did you start the pipeline, so what's you -profile
parameter?
Besides, cloud compute resources are pre-configured here https://github.com/replikation/poreCov/blob/master/configs/nodes.config for each process. Passing options like --cores
will not have an effect for cloud compute. You can also define your own configuration and inject this via -c my_config
, for example when you want to adjust the cores and ram of certain processes.
hi @karthick1087 if you would like i could branch a AWS branch and we implement a generic AWS profile. i only have access to GCP and not AWS so i cant really test this wokflow.
hi @karthick1087 if you would like i could branch a AWS branch and we implement a generic AWS profile. i only have access to GCP and not AWS so i cant really test this wokflow.
Hi thats great!!, If you Could branch for AWS. I will test the the work flow in AWS
@karthick1087 hi can you detail how you are currently starting was runs?
curious about the cores now. Do I have to use --cores to get the pipeline to utilize them? seems like the initialisation parameters do report 0/X cores when i run porecov. how do --cores and --max_cores differ?
Hi!
--cores: Maximum cores used per process --max_cores: Maximum cores run at all
E.g. --cores 4 --max_cores 8 will run at most two 4core processes in parallel. We implemented this to prevent to many processes run in parallel on large work stations with many cores that are also used by others.
However, these params only effect local execution (-profile local). For cloud compute and on hpc (slurp, lsf,...) a pre-configured setup of cores per process is run.
On Tue, 12 Oct 2021, 07:37 omarkr8, @.***> wrote:
curious about the cores now. Do I have to use --cores to get the pipeline to utilize them? seems like the initialisation parameters do report 0/X cores when i run porecov. how do --cores and --max_cores differ?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/replikation/poreCov/issues/154#issuecomment-940680341, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADN2CZ5AR7O4EXLUSCFOYYTUGPCQ5ANCNFSM5DSJG4ZQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Hi, sorry to bother with third consequtive query. I am running the pipeline in the AWS EC2 cloud instance. Even after I supplied with --ncores=16, some of the tools running with single cpu only, others are not engaged. Is there something that i can do to speed up the process.