search

vib-singlecell-nf / vsn-pipelines

A repository of pipelines for single-cell data in Nextflow DSL2
GNU General Public License v3.0
75 stars 32 forks source link

[BUG] Include statement is not allowed within a workflow definition #369

Open matt-sd-watson opened 2 years ago

matt-sd-watson commented 2 years ago

Describe the bug

I am proceeding through the steps as described in the 10k PBMC tutorial: https://vsn-pipelines-examples.readthedocs.io/en/latest/PBMC10k.html

The following error is encountered when attempting to process a single sample: `Launching vib-singlecell-nf/vsn-pipelines [confident_legentil] - revision: 96cff8e597 [master] Include statement is not allowed within a workflow definition

-- Check script '/Users/mattsdwatson/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details`

To Reproduce Steps to reproduce the behavior:

Follow the configuration steps as described here on Macbook.

  1. Run using this entry point:

`Launching vib-singlecell-nf/vsn-pipelines [confident_legentil] - revision: 96cff8e597 [master] Include statement is not allowed within a workflow definition

-- Check script '/Users/mattsdwatson/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details`

  1. See error:

`N E X T F L O W ~ version 21.10.5 Launching vib-singlecell-nf/vsn-pipelines [confident_legentil] - revision: 96cff8e597 [master] Include statement is not allowed within a workflow definition

-- Check script '/Users/mattsdwatson/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details`

Screenshots

The full stack trace output is attached here describing the error in further detail.

log.txt

Please complete the following information:

Note: that Singularity is not installed or used for this test case.

roncoronimiguel commented 2 years ago

+1 on:

and on:

KrisDavie commented 2 years ago

Hey @matt-sd-watson and @roncoronimiguel, could you try again using Nextflow version 21.04.3 and let me know if you have the same issue?

Cheers,

Kris

roncoronimiguel commented 2 years ago

@KrisDavie no issues when using Nexflow 21.04.3

KrisDavie commented 2 years ago

Thanks, looking into it, there was indeed a bug which has been fixed that affects our code on newer versions of nextflow. For now, the solution is to use 21.04.3, and we'll fix this for the next release.

MarcDalod commented 2 years ago

Hi @KrisDavie,

I am still having the issue and it is does not change whatever version of nextflow I use, 

  `me@gauss07$ ./nextflow-21.10.3-all -C pbmc10k.vsn-pipelines.complete.config     run vib-   singlecell-nf/vsn-pipelines     -entry single_sample
   N E X T F L O W  ~  version 21.10.3
   Launching `vib-singlecell-nf/vsn-pipelines` [disturbed_mayer] - revision: 96cff8e597 [master]   
   Missing process or function with name 'include'

   -- Check script '/home/me/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details
   me@gauss07:/mnt/DOSI/MDALAB/USERS/Active/Ammar_Sabir_me/scenic_analysis/example$ vim  /home/me/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf
   me@gauss07$ nextflow -C pbmc10k.vsn-pipelines.complete.config     run vib-singlecell-nf/vsn-pipelines     -entry single_sample
   N E X T F L O W  ~  version 21.10.6
   Launching `vib-singlecell-nf/vsn-pipelines` [sleepy_gates] - revision: 96cff8e597 [master]
   Include statement is not allowed within a workflow definition

   -- Check script '~/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details

 me@gauss07 $ ./nextflow-21.10.4-all -C pbmc10k.vsn-pipelines.complete.config     run vib-singlecell- nf/vsn-pipelines     -entry single_sample
 N E X T F L O W  ~  version 21.10.4
 Launching `vib-singlecell-nf/vsn-pipelines` [romantic_nobel] - revision: 96cff8e597 [master]
 Include statement is not allowed within a workflow definition

 -- Check script '/home/cheema/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details

I tried using version 21.10.6, 21.10.4 and 21.10.3 and it does not remove the error. My config file is given below,

I don't know if this is due to some problem in my config file, please have a look on the config file given below,

        manifest {
       name = 'vib-singlecell-nf/vsn-pipelines'
       description = 'A repository of pipelines for single-cell data in Nextflow DSL2'
       homePage = 'https://github.com/vib-singlecell-nf/vsn-pipelines'
       version = '0.27.0'
       mainScript = 'main.nf'
       defaultBranch = 'master'
       nextflowVersion = '!>=21.04.3'
      }

params {
global {
  project_name = 'Test'
  outdir = 'out'
  species = 'human'
  genome {
     assembly = 'hg38'
  }
  }
  misc {
  test {
     enabled = false
  }
  }
 utils {
  container = 'vibsinglecellnf/utils:0.4.0'
  file_converter {
     off = 'h5ad'
     tagCellWithSampleId = true
     remove10xGEMWell = false
     useFilteredMatrix = true
     makeVarIndexUnique = false
  }
  publish {
     compressionLevel = 6
     annotateWithBatchVariableName = false
     mode = 'copy'
  }
  scope {
     genome = ''
     tree {
        level_1 = ''
        level_2 = ''
        level_3 = ''
     }
  }
 }
 tools {
  scanpy {
     container = 'vibsinglecellnf/scanpy:1.8.1'
     report {
        annotations_to_plot = []
     }
     feature_selection {
        report_ipynb = '/src/scanpy/bin/reports/sc_select_variable_genes_report.ipynb'
        flavor = 'seurat'
        minMean = 0.0125
        maxMean = 3
        minDisp = 0.5
        off = 'h5ad'
     }
     feature_scaling {
        method = 'zscore_scale'
        maxSD = 10
        off = 'h5ad'
     }
     neighborhood_graph {
        nPcs = 50
        off = 'h5ad'
     }
     dim_reduction {
        report_ipynb = '/src/scanpy/bin/reports/sc_dim_reduction_report.ipynb'
        pca {
           method = 'pca'
           nComps = 50
           off = 'h5ad'
        }
        umap {
           method = 'umap'
           off = 'h5ad'
        }
        tsne {
           method = 'tsne'
           off = 'h5ad'
        }
     }
     clustering {
        preflight_checks = true
        report_ipynb = '/src/scanpy/bin/reports/sc_clustering_report.ipynb'
        method = 'louvain'
        resolution = 0.8
        off = 'h5ad'
     }
     marker_genes {
        method = 'wilcoxon'
        ngenes = 0
        groupby = 'louvain'
        off = 'h5ad'
     }
     filter {
        report_ipynb = '/src/scanpy/bin/reports/sc_filter_qc_report.ipynb'
        cellFilterStrategy = 'fixedthresholds'
        cellFilterMinNGenes = 200
        cellFilterMaxNGenes = 4000
        cellFilterMaxPercentMito = 0.15
        geneFilterMinNCells = 3
        off = 'h5ad'
        outdir = 'out'
     }
     data_transformation {
        method = 'log1p'
        off = 'h5ad'
     }
     normalization {
        method = 'cpx'
        targetSum = 10000
        off = 'h5ad'
     }
  }
  scenic {
     container = 'aertslab/pyscenic:0.10.4'
     scenicoutdir = 'out/scenic/'
     filteredLoom = '/ddn1/vol1/staging/leuven/stg_00002/lcb/cflerin/testruns/scenic-nf_testing /expr_mat.loom'
     scenicOutputLoom = 'SCENIC_output.loom'
     scenicScopeOutputLoom = 'SCENIC_SCope_output.loom'
     mode = 'dask_multiprocessing'
     client_or_address = ''
     cell_id_attribute = 'CellID'
     gene_attribute = 'Gene'
     report_ipynb = '/src/scenic/bin/reports/scenic_report.ipynb'
     skipReports = false
     grn {
        algorithm = 'grnboost2'
        tfs = 'allTFs_mm.txt'
     }
     cistarget {
        adj = 'adj.tsv'
        type = ''
        rank_threshold = 5000
        auc_threshold = 0.05
        nes_threshold = 3.0
        min_orthologous_identity = 0.0
        max_similarity_fdr = 0.001
        annotations_fname = ''
        thresholds = '0.75,0.90'
        top_n_targets = 50
        top_n_regulators = '5,10,50'
        min_genes = 20
        all_modules = false
        motifsDb = '/ddn1/vol1/staging/leuven/stg_00002/lcb/cflerin/testruns/scenic-nf_testing/genome-ranking.feather'
        motifsAnnotation = 'motifs-v9-nr.mgi-m0.001-o0.0.tbl'
        tracksDb = '/staging/leuven/res_00001/databases/cistarget/databases/homo_sapiens/hg38/refseq_r80/tc_v1/gene_based/mm10__refseq-r80__*feather'
     }
     aucell {
        output = 'aucell_output.loom'
        rank_threshold = 5000
        auc_threshold = 0.05
        nes_threshold = 3.0
        min_genes_regulon = 5
        min_regulon_gene_occurrence = 5
     }
     numRuns = 100
     aggregate_features {
        use_chunking = true
        output_format = 'csv'
        compression = 'gzip'
     }
     }
     }
     data {
     tenx {
     cellranger_mex = 'data/raw_gene_expression_df.csv'
     }
     }
     }

      process {
      executor = 'local'
      cpus = 2
      memory = '60 GB'
      clusterOptions = '-A cluster_account'
      withLabel:compute_resources__default {
        time = '1h'
       }
      withLabel:compute_resources__minimal {
        cpus = 1
          memory = '1 GB'
        }
          withLabel:compute_resources__mem {
          cpus = 4
          memory = '160 GB'
          }
          withLabel:compute_resources__cpu {
          cpus = 20
          memory = '80 GB'
       }
       withLabel:compute_resources__report {
          maxForks = 2
          cpus = 1
          memory = '160 GB'
       }
       withLabel:compute_resources__24hqueue {
          time = '24h'
       }
       withLabel:'compute_resources__scenic.*' {
          cpus = 4
          memory = '60 GB'
          time = '24h'
          maxForks = 1
       }
       withLabel:compute_resources__scenic_grn {
          cpus = 4
          memory = '120 GB'
          time = '24h'
       }
       withLabel:compute_resources__scenic_cistarget {
          cpus = 4
          memory = '80 GB'
       }
       withLabel:compute_resources__scenic_aucell {
          cpus = 4
          memory = '80 GB'
       }
       withLabel:'compute_resources__scenic_multiruns.*' {
          cpus = 4
          memory = '60 GB'
          time = '24h'
       }
       withLabel:compute_resources__scenic_multiruns_motifs2regulons {
          memory = '60 GB'
       }
    }

    timeline {
       enabled = true
       file = 'out/nextflow_reports/execution_timeline.html'
    }

    report {
       enabled = true
       file = 'out/nextflow_reports/execution_report.html'
    }

    trace {
       enabled = true
       file = 'out/nextflow_reports/execution_trace.txt'
    }

    dag {
       enabled = true
       file = 'out/nextflow_reports/pipeline_dag.svg'
    }

    min {
       enabled = false
    }

    vsc {
       enabled = false
    }memory = '1 GB'
    }
           withLabel:compute_resources__mem {
              cpus = 4
              memory = '160 GB'
           }
           withLabel:compute_resources__cpu {
              cpus = 20
              memory = '80 GB'
           }
           withLabel:compute_resources__report {
              maxForks = 2
              cpus = 1
              memory = '160 GB'
           }
           withLabel:compute_resources__24hqueue {
              time = '24h'
           }
           withLabel:'compute_resources__scenic.*' {
              cpus = 4
              memory = '60 GB'
              time = '24h'
              maxForks = 1
           }
           withLabel:compute_resources__scenic_grn {
              cpus = 4
              memory = '120 GB'
              time = '24h'
           }
           withLabel:compute_resources__scenic_cistarget {
              cpus = 4
              memory = '80 GB'
           }
           withLabel:compute_resources__scenic_aucell {
              cpus = 4
              memory = '80 GB'
           }
           withLabel:'compute_resources__scenic_multiruns.*' {
              cpus = 4
              memory = '60 GB'
              time = '24h'
           }
           withLabel:compute_resources__scenic_multiruns_motifs2regulons {
              memory = '60 GB'
           }
        }

        timeline {
           enabled = true
           file = 'out/nextflow_reports/execution_timeline.html'
        }

        report {
           enabled = true
           file = 'out/nextflow_reports/execution_report.html'
        }

        trace {
           enabled = true
           file = 'out/nextflow_reports/execution_trace.txt'
        }

        dag {
           enabled = true
           file = 'out/nextflow_reports/pipeline_dag.svg'
        }

        min {
           enabled = false
        }

        vsc {
           enabled = false
        }

        singularity {
           enabled = true
           autoMounts = true
           runOptions = '--cleanenv -H $PWD -B ${HOME}'
        }

                singularity {
                   enabled = true
                   autoMounts = true
                   runOptions = '--cleanenv -H $PWD -B ${HOME}'
                }

Thanks,

KrisDavie commented 2 years ago

Hi @MarcDalod, you must use version 21.04.3, anything above 21.04.3 will not work until we update VSN unfortunately, and the versions you mentioned were all 21.10 or higher.

Can you try 21.04.3 and report back?

Cheers,

Kris

MarcDalod commented 2 years ago

Hi @KrisDavie ,

Thanks for your response, I have tried with version 21.04.3 but it is giving the problem of java, but java is already installed, 

      cheemaa@gauss05$ ./nextflow-21.04.3-all 
      ERROR: Cannot find Java or it's a wrong version -- please make sure that Java 8 or later is installed
      NOTE: Nextflow is trying to use the Java VM defined by the following environment variables:
       JAVA_CMD: /usr/bin/java
       JAVA_HOME: 

      $ java --version
      openjdk 17.0.2 2022-01-18
      OpenJDK Runtime Environment (build 17.0.2+8-Ubuntu-120.04)
      OpenJDK 64-Bit Server VM (build 17.0.2+8-Ubuntu-120.04, mixed mode, sharing)

      cheemaa@gauss05$ which java
      /usr/bin/java

Can you please let me know if there is some solution to this problem ?

KrisDavie commented 2 years ago

Hey @MarcDalod,

That version of Java (17) is much to high for the older version of nextflow, you should use something between 8 and 12 as described here: https://github.com/nextflow-io/nextflow/issues/1662

This won't be an issue any longer once we update VSN to use newer nextflow versions as higher java versions are supported in nextflow 21.05 and higher.

Cheers,

Kris

MarcDalod commented 2 years ago

Hi @KrisDavie ,

Then is it possible that I make a docker image with the right version of java, nexflow and the pyscenic container? Do you think it could work this way?

Thanks, Ammar