search

nf-core / scrnaseq

A single-cell RNAseq pipeline for 10X genomics data
https://nf-co.re/scrnaseq
MIT License
214 stars 172 forks source link

Dev branch failing on test config #158

Closed ameynert closed 2 years ago

ameynert commented 2 years ago

Description of the bug

Running the test profile (with conda) on the dev branch fails on process MTX_TO_H5AD because files in *Solo/Gene/filtered are not generated by STAR_ALIGN.

Command used and terminal output

nextflow run nf-core/scrnaseq -profile test,conda --outdir . -r dev
N E X T F L O W  ~  version 22.04.5
Launching `https://github.com/nf-core/scrnaseq` [festering_albattani] DSL2 - revision: f580beb2f6 [dev]

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/scrnaseq v2.1.1dev
------------------------------------------------------
Core Nextflow options
  revision                  : dev
  runName                   : festering_albattani
  launchDir                 : /Users/ameyner2/Documents/GitHub/scratch
  workDir                   : /Users/ameyner2/Documents/GitHub/scratch/work
  projectDir                : /Users/ameyner2/.nextflow/assets/nf-core/scrnaseq
  userName                  : ameyner2
  profile                   : test,conda
  configFiles               : /Users/ameyner2/.nextflow/assets/nf-core/scrnaseq/nextflow.config

Input/output options
  input                     : https://github.com/nf-core/test-datasets/raw/scrnaseq/samplesheet-2-0.csv
  outdir                    : .

Mandatory arguments
  aligner                   : star

Reference genome options
  fasta                     : https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/GRCm38.p6.genome.chr19.fa
  gtf                       : https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/gencode.vM19.annotation.chr19.gtf

Kallisto/BUS Options
  bustools_correct          : true

Institutional config options
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

Generic options
  enable_conda              : true

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/scrnaseq for your analysis please cite:

* The pipeline
  https://doi.org/10.5281/zenodo.3568187

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/scrnaseq/blob/master/CITATIONS.md
------------------------------------------------------
executor >  local (2)
executor >  local (2)
executor >  local (2)
executor >  local (4)
executor >  local (7)
executor >  local (7)
executor >  local (11)
[2a/785748] process > NFCORE_SCRNASEQ:SCRNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet-2-0.csv)      [100%] 1 of 1 ✔
[d6/5e2e8f] process > NFCORE_SCRNASEQ:SCRNASEQ:FASTQC_CHECK:FASTQC (Sample_Y)                           [100%] 2 of 2 ✔
executor >  local (11)
[2a/785748] process > NFCORE_SCRNASEQ:SCRNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet-2-0.csv)      [100%] 1 of 1 ✔
[d6/5e2e8f] process > NFCORE_SCRNASEQ:SCRNASEQ:FASTQC_CHECK:FASTQC (Sample_Y)                           [100%] 2 of 2 ✔
[97/7cf8bb] process > NFCORE_SCRNASEQ:SCRNASEQ:GTF_GENE_FILTER (GRCm38.p6.genome.chr19.fa)              [100%] 1 of 1 ✔
[bd/f9e26e] process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_GENOMEGENERATE (GRCm38.p6.genome.chr19.fa) [100%] 1 of 1 ✔
executor >  local (11)
[2a/785748] process > NFCORE_SCRNASEQ:SCRNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet-2-0.csv)      [100%] 1 of 1 ✔
[d6/5e2e8f] process > NFCORE_SCRNASEQ:SCRNASEQ:FASTQC_CHECK:FASTQC (Sample_Y)                           [100%] 2 of 2 ✔
[97/7cf8bb] process > NFCORE_SCRNASEQ:SCRNASEQ:GTF_GENE_FILTER (GRCm38.p6.genome.chr19.fa)              [100%] 1 of 1 ✔
[bd/f9e26e] process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_GENOMEGENERATE (GRCm38.p6.genome.chr19.fa) [100%] 1 of 1 ✔
[1a/1812e8] process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_ALIGN (Sample_Y)                           [100%] 2 of 2 ✔
[7e/14da29] process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_Y)                    [100%] 2 of 2, failed: 2 ✘
[-        ] process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:CONCAT_H5AD                               -
[-        ] process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT                             -
[5d/6652e2] process > NFCORE_SCRNASEQ:SCRNASEQ:CUSTOM_DUMPSOFTWAREVERSIONS (1)                          [100%] 1 of 1 ✔
[e9/909aa3] process > NFCORE_SCRNASEQ:SCRNASEQ:MULTIQC                                                  [100%] 1 of 1 ✔
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/scrnaseq] Pipeline completed with errors-
Error executing process > 'NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_X)'

Caused by:
  Process `NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_X)` terminated with an error exit status (1)

Command executed:

  # convert file types
  mtx_to_h5ad.py \
      --aligner star \
      --sample Sample_X \
      --mtx *.Solo.out/Gene*/filtered/matrix.mtx.gz \
      --barcode *.Solo.out/Gene*/filtered/barcodes.tsv.gz \
      --feature *.Solo.out/Gene*/filtered/features.tsv.gz \
      --out Sample_X_matrix.h5ad

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File "/Users/ameyner2/.nextflow/assets/nf-core/scrnaseq/bin/mtx_to_h5ad.py", line 45, in <module>
      adata = mtx_to_adata(
    File "/Users/ameyner2/.nextflow/assets/nf-core/scrnaseq/bin/mtx_to_h5ad.py", line 19, in mtx_to_adata
      adata = sc.read_mtx(mtx_file)
    File "/Users/ameyner2/Documents/GitHub/scratch/work/conda/env-8ceee76ce5640a76fa8925a8ce8f49a3/lib/python3.10/site-packages/anndata/_io/read.py", line 317, in read_mtx
      X = mmread(fspath(filename)).astype(dtype)
    File "/Users/ameyner2/Documents/GitHub/scratch/work/conda/env-8ceee76ce5640a76fa8925a8ce8f49a3/lib/python3.10/site-packages/scipy/io/_mmio.py", line 77, in mmread
      return MMFile().read(source)
    File "/Users/ameyner2/Documents/GitHub/scratch/work/conda/env-8ceee76ce5640a76fa8925a8ce8f49a3/lib/python3.10/site-packages/scipy/io/_mmio.py", line 435, in read
      stream, close_it = self._open(source)
    File "/Users/ameyner2/Documents/GitHub/scratch/work/conda/env-8ceee76ce5640a76fa8925a8ce8f49a3/lib/python3.10/site-packages/scipy/io/_mmio.py", line 324, in _open
      stream = gzip.open(filespec, mode)
    File "/Users/ameyner2/Documents/GitHub/scratch/work/conda/env-8ceee76ce5640a76fa8925a8ce8f49a3/lib/python3.10/gzip.py", line 58, in open
      binary_file = GzipFile(filename, gz_mode, compresslevel)
    File "/Users/ameyner2/Documents/GitHub/scratch/work/conda/env-8ceee76ce5640a76fa8925a8ce8f49a3/lib/python3.10/gzip.py", line 174, in __init__
      fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
  FileNotFoundError: [Errno 2] No such file or directory: '*.Solo.out/Gene*/filtered/matrix.mtx.gz'

Work dir:
  /Users/ameyner2/Documents/GitHub/scratch/work/3f/5610298bacfd905acaa1bf3ceca10a

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

Relevant files

No response

System information

apeltzer commented 2 years ago

Is this conda specific or same happens with docker/singularity too?

ameynert commented 2 years ago

@ogibson is trying with Docker.

ogibson commented 2 years ago

No issues with STAR/docker on the dev branch.

ameynert commented 2 years ago

STAR/singularity also looks fine - just ran it on my uni HPC.

ameynert commented 2 years ago

Conda has worked on my uni HPC. Mac/Conda-specific issue?

apeltzer commented 2 years ago

Quite likely then. M1 mac?

We also kind of tell people in some pipelines not to use conda anymore due to some low level issues like this ...

ameynert commented 2 years ago

Not M1, but agree on not using conda in production! I'll close this since it's unlikely to reproduce.