search

alesssia / YAMP

YAMP: Yet Another Metagenomic Pipeline
GNU General Public License v3.0
56 stars 27 forks source link

Error: `Missing process or function` when running the pipeline #29

Open jacodela opened 4 months ago

jacodela commented 4 months ago

Hi, thanks for developing and mantaining YAMP!

I've been trying to run YAMP from a fresh installation using singularity but I keep getting this error, regardless of whether I use the base or the test profiles using the test samples

Missing process or function call([/mnt/lustre/groups/maier/maide581/bin/YAMP/conf/multiqc_config.yaml, /mnt/lustre/groups/maier/maide581/bin/YAMP/work/workflow_summary_mqc.yaml, DataflowVariable(value=null), DataflowVariable(value=null), DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?]])

I have configured the nextflow.config file to work with singularity as specified in the wiki as follows:

singularity {
  singularity.enabled = true
  singularity.cacheDir = '/mnt/lustre/groups/maier/maide581/bin/YAMP/singularity/'
  process.executor = 'slurm'
}

The complete nextflow log is

May-07 16:31:09.476 [main] DEBUG nextflow.cli.Launcher - $> nextflow run YAMP.nf --reads1 /mnt/lustre/groups/maier/maide581/bin/YAMP/data/test_data/random_ncbi_reads_with_duplicated_and_contaminants_R1.fastq.gz --reads2 /mnt/lustre/groups/maier/maide581/bin/YAMP/data/test_data/random_ncbi_reads_with_duplicated_and_contaminants_R1.fastq.gz --prefix TST --outdir ./output_folder --mode complete -profile test,singularity
May-07 16:31:09.623 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.10.1
May-07 16:31:09.638 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/mnt/lustre/home/maier/maide581/.nextflow/plugins; core-plugins: nf-amazon@2.1.4,nf-azure@1.3.3,nf-cloudcache@0.3.0,nf-codecommit@0.1.5,nf-console@1.0.6,nf-ga4gh@1.1.0,nf-google@1.8.3,nf-tower@1.6.3,nf-wave@1.0.1
May-07 16:31:09.647 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
May-07 16:31:09.648 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
May-07 16:31:09.650 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
May-07 16:31:09.660 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
May-07 16:31:09.674 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /mnt/lustre/groups/maier/maide581/bin/YAMP/nextflow.config
May-07 16:31:09.675 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /mnt/lustre/groups/maier/maide581/bin/YAMP/nextflow.config
May-07 16:31:09.697 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `test,singularity`
May-07 16:31:10.860 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [test, test_single_end_QC, test_single_end_characterisation, test_paired_end_complete, test_paired_end_characterisation, test_paired_end_QC_nodedup, test_single_end_complete, docker, test_paired_end_QC, create, awsbatch, singularity, test_single_end_QC_nodedup, base]
May-07 16:31:10.931 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
May-07 16:31:10.949 [main] INFO  nextflow.cli.CmdRun - Launching `YAMP.nf` [nice_pauling] DSL2 - revision: 95d5c279c2
May-07 16:31:10.950 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
May-07 16:31:10.950 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
May-07 16:31:10.957 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /mnt/lustre/home/maier/maide581/.nextflow/secrets/store.json
May-07 16:31:10.960 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@3f183caa] - activable => nextflow.secret.LocalSecretsProvider@3f183caa
May-07 16:31:11.002 [main] DEBUG nextflow.Session - Session UUID: 1d3230df-46db-4daa-9f68-cec2caa6f111
May-07 16:31:11.002 [main] DEBUG nextflow.Session - Run name: nice_pauling
May-07 16:31:11.002 [main] DEBUG nextflow.Session - Executor pool size: 16
May-07 16:31:11.013 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
May-07 16:31:11.037 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 23.10.1 build 5891
  Created: 12-01-2024 22:01 UTC (23:01 CEST)
  System: Linux 5.14.0-362.24.1.el9_3.0.1.x86_64
  Runtime: Groovy 3.0.19 on OpenJDK 64-Bit Server VM 21-internal-adhoc.conda.src
  Encoding: UTF-8 (UTF-8)
  Process: 284017@login1 [127.0.0.1]
  CPUs: 16 - Mem: 503.2 GB (482.2 GB) - Swap: 0 (0) - Virtual threads ON
May-07 16:31:11.074 [main] DEBUG nextflow.Session - Work-dir: /mnt/lustre/groups/maier/maide581/bin/YAMP/work [lustre]
May-07 16:31:11.094 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
May-07 16:31:11.108 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
May-07 16:31:11.152 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
May-07 16:31:11.162 [main] DEBUG nextflow.util.CustomPoolFactory - Creating virtual thread pool
May-07 16:31:11.344 [main] DEBUG nextflow.Session - Session start
May-07 16:31:11.831 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
May-07 16:31:12.550 [main] INFO  nextflow.Nextflow - ---------------------------------------------
YET ANOTHER METAGENOMIC PIPELINE (YAMP) 
---------------------------------------------

Analysis introspection:

May-07 16:31:12.555 [main] INFO  nextflow.Nextflow - Starting time              : Tue May 07 16:31:12 CEST 2024
Environment                : 
Pipeline Name              : YAMP
Pipeline Version           : 0.9.6.0
Config Profile             : test,singularity
Resumed                    : false
Nextflow version           : 23.10.1 build 5891 (12-01-2024 22:01 UTC)
Java version               : 21-internal
Java Virtual Machine       : OpenJDK 64-Bit Server VM(21-internal-adhoc.conda.src)
Operating system           : Linux amd64 v5.14.0-362.24.1.el9_3.0.1.x86_64
User name                  : maide581
Container Engine           : singularity
Container                  : [:]
BBmap                      : https://depot.galaxyproject.org/singularity/bbmap:38.87--h1296035_0
FastQC                     : https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0
biobakery                  : biobakery/workflows:3.0.0.a.6.metaphlanv3.0.7
qiime                      : qiime2/core:2020.8
MultiQC                    : https://depot.galaxyproject.org/singularity/multiqc:1.9--py_1
Running parameters         : 
Reads                      : [/mnt/lustre/groups/maier/maide581/bin/YAMP/data/test_data/random_ncbi_reads_with_duplicated_and_contaminants_R1.fastq.gz, /mnt/lustre/groups/maier/maide581/bin/YAMP/data/test_data/random_ncbi_reads_with_duplicated_and_contaminants_R1.fastq.gz]
Prefix                     : TST
Running mode               : complete
Layout                     : Paired-End
Performing de-duplication  : true
Synthetic contaminants     : 
Artefacts                  : /mnt/lustre/groups/maier/maide581/bin/YAMP/assets/data/sequencing_artifacts.fa.gz
Phix174ill                 : /mnt/lustre/groups/maier/maide581/bin/YAMP/assets/data/phix174_ill.ref.fa.gz
Adapters                   : /mnt/lustre/groups/maier/maide581/bin/YAMP/assets/data/adapters.fa
Trimming parameters        : 
Input quality offset       : ASCII+33
Min phred score            : 10
Min length                 : 60
kmer lenght                : 23
Shorter kmer               : 11
Max Hamming distance       : 1
Decontamination parameters : 
Contaminant (pan)genome    : /mnt/lustre/groups/maier/maide581/bin/YAMP/assets/demo/genome.fa
Min alignment identity     : 0.95
Max indel length           : 3
Max alignment band         : 0.16
MetaPhlAn parameters       : 
MetaPhlAn database         : /mnt/lustre/groups/maier/maide581/bin/YAMP/assets/data/metaphlan_databases/
Bowtie2 options            : very-sensitive
HUMAnN parameters          : 
Chocophlan database        : /mnt/lustre/groups/maier/maide581/bin/YAMP/assets/demo/chocophlan
Uniref database            : /mnt/lustre/groups/maier/maide581/bin/YAMP/assets/demo/uniref
Folders                    : 
Output dir                 : ./output_folder
Working dir                : /mnt/lustre/groups/maier/maide581/bin/YAMP/work
Script dir                 : /mnt/lustre/groups/maier/maide581/bin/YAMP
Lunching dir               : /mnt/lustre/groups/maier/maide581/bin/YAMP
May-07 16:31:12.557 [main] INFO  nextflow.Nextflow - 
May-07 16:31:12.687 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:get_software_versions` matches process get_software_versions
May-07 16:31:12.690 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.690 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.695 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
May-07 16:31:12.701 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=16; memory=503.2 GB; capacity=16; pollInterval=100ms; dumpInterval=5m
May-07 16:31:12.703 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
May-07 16:31:12.774 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:dedup` matches process dedup
May-07 16:31:12.777 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.777 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.801 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:remove_synthetic_contaminants` matches process remove_synthetic_contaminants
May-07 16:31:12.812 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.812 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.817 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:trim` matches process trim
May-07 16:31:12.818 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.818 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.822 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:index_foreign_genome` matches process index_foreign_genome
May-07 16:31:12.823 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.823 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.830 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:decontaminate` matches process decontaminate
May-07 16:31:12.831 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.831 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.836 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:quality_assessment` matches process quality_assessment
May-07 16:31:12.837 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.837 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.841 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:profile_taxa` matches process profile_taxa
May-07 16:31:12.842 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.842 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.847 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:profile_function` matches process profile_function
May-07 16:31:12.847 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.847 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.851 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:alpha_diversity` matches process alpha_diversity
May-07 16:31:12.852 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
May-07 16:31:12.852 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
May-07 16:31:12.879 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_dcbca6b588780e9b: /mnt/lustre/groups/maier/maide581/bin/YAMP/./modules/house_keeping.nf
  Script_15717f98cab785a5: /mnt/lustre/groups/maier/maide581/bin/YAMP/./modules/quality_control.nf
  Script_ee9478e9ccceff0e: /mnt/lustre/groups/maier/maide581/bin/YAMP/YAMP.nf
  Script_0bf69ef226688eaa: /mnt/lustre/groups/maier/maide581/bin/YAMP/./modules/community_characterisation.nf
May-07 16:31:12.880 [main] DEBUG nextflow.Session - Session aborted -- Cause: Missing process or function call([/mnt/lustre/groups/maier/maide581/bin/YAMP/conf/multiqc_config.yaml, /mnt/lustre/groups/maier/maide581/bin/YAMP/work/workflow_summary_mqc.yaml, DataflowVariable(value=null), DataflowVariable(value=null), DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?]])
May-07 16:31:12.893 [main] DEBUG nextflow.Session - The following nodes are still active:
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] mix
  [operator] collect
  [operator] ifEmpty
  [operator] ifEmpty
  [operator] ifEmpty
  [operator] ifEmpty
  [operator] ifEmpty
  [operator] ifEmpty
  [operator] ifEmpty
  [operator] ifEmpty
  [operator] ifEmpty
  [operator] ifEmpty

May-07 16:31:12.898 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
May-07 16:31:12.903 [main] ERROR nextflow.cli.Launcher - @unknown
nextflow.exception.MissingProcessException: Missing process or function call([/mnt/lustre/groups/maier/maide581/bin/YAMP/conf/multiqc_config.yaml, /mnt/lustre/groups/maier/maide581/bin/YAMP/work/workflow_summary_mqc.yaml, DataflowVariable(value=null), DataflowVariable(value=null), DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?], DataflowBroadcast around DataflowStream[?]])
    at nextflow.script.WorkflowDef.run(WorkflowDef.groovy:191)
    at nextflow.script.BindableDef.invoke_a(BindableDef.groovy:51)
    at nextflow.script.IterableDef$invoke_a.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
    at nextflow.script.BaseScript.run0(BaseScript.groovy:183)
    at nextflow.script.BaseScript.run(BaseScript.groovy:192)
    at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
    at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:242)
    at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:137)
    at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
    at nextflow.cli.Launcher.run(Launcher.groovy:500)
    at nextflow.cli.Launcher.main(Launcher.groovy:672)
Caused by: groovy.lang.MissingMethodException: No signature of method: ch.qos.logback.classic.Logger.call() is applicable for argument types: (sun.nio.fs.UnixPath, sun.nio.fs.UnixPath, groovyx.gpars.dataflow.DataflowVariable...) values: [/mnt/lustre/groups/maier/maide581/bin/YAMP/conf/multiqc_config.yaml, ...]
Possible solutions: wait(), any()
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:70)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
    at Script_ee9478e9ccceff0e$_runScript_closure3$_closure4.doCall(Script_ee9478e9ccceff0e:516)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030)
    at groovy.lang.Closure.call(Closure.java:427)
    at groovy.lang.Closure.call(Closure.java:406)
    at nextflow.script.WorkflowDef.run0(WorkflowDef.groovy:204)
    at nextflow.script.WorkflowDef.run(WorkflowDef.groovy:188)
    ... 13 common frames omitted
alesssia commented 4 months ago

Hi @jacodela , I may have found the problem, running some tests and, if all goes well, in the next few days I'll push an update! Thanks for your patience 🙏

jacodela commented 2 months ago

Hi, is there a solution in sight? Alternatively, is there a way to use a previous version of YAMP to see if the problem doesn't show up?