Open SomePersonSomeWhereInTheWorld opened 4 months ago
@kirilenkobm Could you pls have a look if the --cluster_parameters is a retired parameter? Thx
Here is the top part of the .nextflow.log
. Is there another option I need to use?
Apr-30 12:32:43.670 [main] DEBUG nextflow.cli.Launcher - $> nextflow /path/to/me/make_lastz_chains/parallelization/execute_joblist.nf --joblist /path/to/me/make_lastz_chains/test_out_1/temp_lastz_run/lastz_joblist.txt -c /path/to/me/make_lastz_chains/test_out_1/temp_lastz_run/lastz_config.nf
Apr-30 12:32:43.723 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 23.10.1
Apr-30 12:32:43.740 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/cluster/home/me/.nextflow/plugins; core-plugins: nf-amazon@2.1.4,nf-azure@1.3.3,nf-cloudcache@0.3.0,nf-codecommit@0.1.5,nf-console@1.0.6,nf-ga4gh@1.1.0,nf-google@1.8.3,nf-tower@1.6.3,nf-wave@1.0.1
Apr-30 12:32:43.749 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Apr-30 12:32:43.750 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Apr-30 12:32:43.752 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Apr-30 12:32:43.766 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Apr-30 12:32:43.784 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /path/to/me/make_lastz_chains/test_out_1/temp_lastz_run/lastz_config.nf
Apr-30 12:32:43.785 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /path/to/me/make_lastz_chains/test_out_1/temp_lastz_run/lastz_config.nf
Apr-30 12:32:43.804 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Apr-30 12:32:44.203 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Apr-30 12:32:44.218 [main] INFO nextflow.cli.CmdRun - Launching `/path/to/me/make_lastz_chains/parallelization/execute_joblist.nf` [ridiculous_mcnulty] DSL2 - revision: 0483b29723
Apr-30 12:32:44.219 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Apr-30 12:32:44.219 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Apr-30 12:32:44.227 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /cluster/home/me/.nextflow/secrets/store.json
Apr-30 12:32:44.230 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@10f7c76] - activable => nextflow.secret.LocalSecretsProvider@10f7c76
Apr-30 12:32:44.275 [main] DEBUG nextflow.Session - Session UUID: 5fae7dbe-8c74-4805-926b-aa6223f5ae87
Apr-30 12:32:44.275 [main] DEBUG nextflow.Session - Run name: ridiculous_mcnulty
Apr-30 12:32:44.276 [main] DEBUG nextflow.Session - Executor pool size: 24
Apr-30 12:32:44.282 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Apr-30 12:32:44.285 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=72; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Apr-30 12:32:44.382 [main] DEBUG nextflow.cli.CmdRun -
Version: 23.10.1 build 5891
Created: 12-01-2024 22:01 UTC (17:01 EDT)
System: Linux 4.18.0-193.el8.x86_64
Runtime: Groovy 3.0.19 on Java HotSpot(TM) 64-Bit Server VM 20.0.1+9-29
Encoding: UTF-8 (UTF-8)
Process: 322176@g261 [10.197.17.16]
CPUs: 24 - Mem: 50 GB (47.8 GB) - Swap: 0 (0)
Apr-30 12:32:44.424 [main] DEBUG nextflow.Session - Work-dir: /path/to/me/make_lastz_chains/test_out_1/temp_lastz_run/work [lustre]
Apr-30 12:32:44.424 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /path/to/me/make_lastz_chains/parallelization/bin
Apr-30 12:32:44.434 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Apr-30 12:32:44.442 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Apr-30 12:32:44.458 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Apr-30 12:32:44.468 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 25; maxThreads: 1000
Apr-30 12:32:44.551 [main] DEBUG nextflow.Session - Session start
Apr-30 12:32:44.692 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Apr-30 12:32:44.801 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Apr-30 12:32:44.802 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Apr-30 12:32:44.809 [main] DEBUG nextflow.executor.Executor - [warm up] executor > slurm
Apr-30 12:32:44.814 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'slurm' > capacity: 1000; pollInterval: 5s; dumpInterval: 5m
Apr-30 12:32:44.816 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: slurm)
Apr-30 12:32:44.817 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m
Apr-30 12:32:44.869 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: execute_jobs
Apr-30 12:32:44.869 [main] DEBUG nextflow.Session - Igniting dataflow network (2)
Apr-30 12:32:44.874 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > execute_jobs
Apr-30 12:32:44.874 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_f6c411a586096bcb: /path/to/me/make_lastz_chains/parallelization/execute_joblist.nf
Apr-30 12:32:44.874 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Apr-30 12:32:44.874 [main] DEBUG nextflow.Session - Session await
Apr-30 12:32:45.049 [Task submitter] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=execute_jobs (5); work-dir=/path/to/me/make_lastz_chains/test_out_1/temp_lastz_run/work/4e/e79fd76c40079431a8db2ed4875930
error [nextflow.exception.ProcessFailedException]: Error submitting process 'execute_jobs (5)' for execution
Apr-30 12:32:45.057 [Task submitter] INFO nextflow.processor.TaskProcessor - [4e/e79fd7] NOTE: Error submitting process 'execute_jobs (5)' for execution -- Execution is retried (1)
Apr-30 12:32:45.091 [Task submitter] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=execute_jobs (1); work-dir=/path/to/me/make_lastz_chains/test_out_1/temp_lastz_run/work/3d/0dd6dde2634f3b7abf79184db82243
@MichaelHiller am I understanding the documentation correctly?
To run the pipeline on a Slurm cluster, for instance, add the
--executor slurm
option. Refer to the Nextflow documentation for a list of supported executors.
The Nextflow Slurm page says:
To enable the SLURM executor, set
process.executor = 'slurm'
in thenextflow.config
file. Resource requests and other job characteristics can be controlled via the following process directives: clusterOptions cpus memory queue time
I know --cluster_executor slurm
works. So if in an interactive or non-interactive, i.e., SBATCH
, job, if --ntasks
is specified, does make_chains.py
consider Slurm options as noted in the Nextlow docs?
FWIW I do not see --cluster_parameters
on the Full list of the pipeline CLI parameters
I'm trying to help a researcher speed up
make_chains.py
results for a mammal. Using this closed issue regarding parallelization, as inspiration, we'd like to speed up the process via our Slurm cluster running RHEL 8. I tried requesting a node via an interactive srun session and starting with 16 CPU with--ntasks
and-c
. Using--executor local
as suggested in the closed thread was painfully slow. The user there mention--cluster_parameters
but that results in:make_chains.py: error: unrecognized arguments: --cluster_parameters cpus=16
May I request assistance here to get the correct syntax?
P.S.. I can confirm the suggested shabang fix in this thread also works to start the sample jobs.