search

nf-core / scrnaseq

A single-cell RNAseq pipeline for 10X genomics data
https://nf-co.re/scrnaseq
MIT License
214 stars 172 forks source link

AssertionError #372

Closed JosephAgim closed 1 month ago

JosephAgim commented 1 month ago

Description of the bug

filename pattern claims to be incorrect but the input on the design.csv file matches with the protocol guide and the fileneames was changed to match the names of the design.csv file

We have tried to add R1 and R2 at the end of the file names and we have also replaced the fastq file names numerical order to instead of having 1, it becomes 001 the only thing we have not tried yet is adding the _S1_L, but since the Error code does not send any output on that. Thus we focused only on the R1 and R2 and the order, example: either 001_R1 and 001_R2 or R1_001 and R2_001 both returns the same AssertionError

Command used and terminal output

Command used:
#!/bin/bash -l
#SBATCH -A naiss2023-22-922       # Project ID (adjust if needed)
#SBATCH --time=50:00:00           # Time limit in HH:MM:SS
#SBATCH -n 10                      # Number of cores
#SBATCH -J scRNAseq_analysis       # Job name
#SBATCH -e error_log.txt           # Error log
#SBATCH -o output_log.txt          # Output log
#SBATCH --mail-type=ALL            # Send email notifications
#SBATCH --mail-user=josag897@student.liu.se  # Email to notify

# Load required modules
module load bioinfo-tools
module load Nextflow/

export NXF_SINGULARITY_CACHEDIR=/proj/naiss2024-23-81/private/vathsa/sc/PD/singularity-images
export NXF_OPTS='-Xms1g -Xmx4g'
export NXF_HOME=/proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home
export NXF_TEMP=${SNIC_TMP:-$PATH}

# Run nf-core/scrnaseq pipeline
nextflow run nf-core/scrnaseq \
  --protocol 10XV2 \
  --input /proj/naiss2024-23-81/private/vathsa/sc/PD/scSample_sheet2.csv \
  --fasta /sw/data/iGenomes/Homo_sapiens/NCBI/GRCh38/Sequence/WholeGenomeFasta/genome.fa \
  --gtf /sw/data/iGenomes/Homo_sapiens/NCBI/GRCh38/Annotation/Genes.gencode/genes.gtf \
  --outdir /proj/naiss2024-23-81/private/vathsa/sc/PD/results \
  -profile uppmax \
  --project naiss2023-22-922 \
  --aligner cellranger \
  -resume
  --multiqc_title "snRNA-seq Analysis"

Terminal output:
# Match R1 in the filename, but only if it is followed by a non-digit or non-character
  # match "file_R1.fastq.gz", "file.R1_000.fastq.gz", etc. but
  # do not match "SRR12345", "file_INFIXR12", etc
  filename_pattern = r"([^a-zA-Z0-9])R1([^a-zA-Z0-9])"

  for i, (r1, r2) in enumerate(chunk_iter(fastqs, 2), start=1):
      # double escapes are required because nextflow processes this python 'template'
      if re.sub(filename_pattern, r"\1R2\2", r1.name) != r2.name:
          raise AssertionError(
              dedent(
                  f"""                We expect R1 and R2 of the same sample to have the same filename except for R1/R2.
                  This has been checked by replacing "R1" with "R2" in the first filename and comparing it to the second filename.
                  If you believe this check shouldn't have failed on your filenames, please report an issue on GitHub!

                  Files involved:
                      - {r1}
                      - {r2}
                  """
              )
          )
      r1.rename(fastq_all / f"{sample_id}_S1_L{i:03d}_R1_001.fastq.gz")
      r2.rename(fastq_all / f"{sample_id}_S1_L{i:03d}_R2_001.fastq.gz")

Relevant files

design.csv content: sample,fastq_1,fastq_2 SRR26129865_1,/proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R1_001.fastq.gz,/proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz

the file names: SRX21843315_SRR26129865_R1_001.fastq.gz SRX21843315_SRR26129865_R2_001.fastq.gz

System information

Nextflow version 2.7.1 Hardware: Uppmax cluster Executor: Slurm Container engine: Singularity Version of nf-core/scrnaseq: 2.7.1

grst commented 1 month ago

Hi,

thanks for reporting! Could you dig out the full .command.log file for the failed process and share it here? I'm a bit confused, because the "terminal output" is just a code snippet from the script, not an actual error message.

In theory, it shouldn't even be necessary to change the filenames. The reason why we implemented this feature in the first place is that input filenames don't need to follow the 10x conventions.

JosephAgim commented 1 month ago

Yes I had problems downloading the file from the server since my PI closed on 1 October yesterday. but i was able to get the contents from the file and i pasted it all into a txt format. I suspect that it is the .nextflow.log file you want?

Den ons 2 okt. 2024 kl 10:04 skrev Gregor Sturm @.***>:

Hi,

thanks for reporting! Could you dig out the full .command.log file for the failed process and share it here? I'm a bit confused, because the "terminal output" is just a code snippet from the script, not an actual error message.

In theory, it shouldn't even be necessary to change the filenames. The reason why we implemented this feature in the first place is that input filenames don't need to follow the 10x conventions.

— Reply to this email directly, view it on GitHub https://github.com/nf-core/scrnaseq/issues/372#issuecomment-2387863231, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUGZAPSW5OD63BCNFN52QBDZZOSIJAVCNFSM6AAAAABO6X6AXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBXHA3DGMRTGE . You are receiving this because you authored the thread.Message ID: @.***>

Sep-26 13:21:18.258 [main] DEBUG nextflow.cli.Launcher - $> nextflow run nf-core/scrnaseq --protocol 10XV2 --input /proj/naiss2024-23-81/private/vathsa/sc/PD/scSample_sheet2.csv --fasta /sw/data/iGenomes/Homo_sapiens/NCBI/GRCh38/Sequence/WholeGenomeFasta/genome.fa --gtf /sw/data/iGenomes/Homo_sapiens/NCBI/GRCh38/Annotation/Genes.gencode/genes.gtf --outdir /proj/naiss2024-23-81/private/vathsa/sc/PD/results -profile uppmax --project naiss2023-22-922 --aligner cellranger -resume Sep-26 13:21:18.638 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 24.04.4 Sep-26 13:21:18.697 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/sw/bioinfo/Nextflow/latest/rackham/nxf_home/plugins; core-plugins: @.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*. Sep-26 13:21:18.730 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: [] Sep-26 13:21:18.733 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: [] Sep-26 13:21:18.739 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode Sep-26 13:21:18.762 [main] INFO org.pf4j.AbstractPluginManager - No plugins Sep-26 13:21:18.790 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: /proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home/scm Sep-26 13:21:20.366 [main] DEBUG nextflow.scm.AssetManager - Git config: /proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home/assets/nf-core/scrnaseq/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/scrnaseq.git Sep-26 13:21:20.414 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory] Sep-26 13:21:20.432 [main] DEBUG nextflow.scm.AssetManager - Git config: /proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home/assets/nf-core/scrnaseq/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/scrnaseq.git Sep-26 13:21:21.696 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home/assets/nf-core/scrnaseq/nextflow.config Sep-26 13:21:21.700 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home/assets/nf-core/scrnaseq/nextflow.config Sep-26 13:21:21.714 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home/secrets/store.json Sep-26 13:21:21.719 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: @. - activable => @. Sep-26 13:21:21.738 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: uppmax Sep-26 13:21:24.599 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [bih, cfc_dev, uzl_omics, ifb_core, embl_hd, denbi_qbic, alice, mjolnir_globe, uppmax, incliva, ilifu, uge, rosalind_uge, lugh, mccleary, unibe_ibu, vai, czbiohub_aws, jax, roslin, ccga_med, tes, scw, unc_longleaf, tigem, tubingen_apg, google, apollo, ipop_up, vsc_calcua, pdc_kth, test_cellrangermulti, googlels, daisybio, eddie, medair, biowulf, apptainer, bi, bigpurple, adcra, cedars, pawsey_setonix, vsc_kul_uhasselt, pawsey_nimbus, ucl_myriad, utd_ganymede, charliecloud, icr_davros, ceres, munin, arm, rosalind, hasta, cfc, uzh, ebi_codon_slurm, ebc, ccga_dx, crick, ku_sund_danhead, marvin, biohpc_gen, shifter, mana, mamba, york_viking, unc_lccc, wehi, awsbatch, wustl_htcf, arcc, imperial, maestro, software_license, genotoul, nci_gadi, abims, janelia, nu_genomics, googlebatch, oist, sahmri, mpcdf, leicester, vsc_ugent, create, sage, cambridge, jex, podman, ebi_codon, cheaha, xanadu, nyu_hpc, test, marjorie, computerome, ucd_sonic, test_multiome, seg_globe, sanger, dkfz, bluebear, pasteur, einstein, ethz_euler, m3c, test_full, imb, ucl_cscluster, tuos_stanage, azurebatch, hki, crukmi, devel, csiro_petrichor, qmul_apocrita, wave, docker, engaging, gis, hypatia, psmn, eva, nygc, fgcz, conda, crg, singularity, mpcdf_viper, self_hosted_runner, tufts, uw_hyak_pedslabs, debug, genouest, cbe, phoenix, gitpod, seawulf, uod_hpc, fub_curta, uct_hpc, aws_tower, binac] Sep-26 13:21:24.674 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration Sep-26 13:21:24.675 [main] DEBUG nextflow.cli.CmdRun - Launching https://github.com/nf-core/scrnaseq [elegant_mayer] DSL2 - revision: 4171377f40 [master] Sep-26 13:21:24.677 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins @. Sep-26 13:21:24.677 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[] Sep-26 13:21:24.677 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved @. Sep-26 13:21:24.678 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-validation version: 1.1.4 Sep-26 13:21:24.701 [main] INFO org.pf4j.AbstractPluginManager - Plugin @.' resolved Sep-26 13:21:24.701 [main] INFO org.pf4j.AbstractPluginManager - Start plugin @.' Sep-26 13:21:24.859 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started @. Sep-26 13:21:24.976 [main] DEBUG nextflow.Session - Session UUID: 99e99620-6710-437f-88ff-da55919e86ad Sep-26 13:21:24.977 [main] DEBUG nextflow.Session - Run name: elegant_mayer Sep-26 13:21:24.977 [main] DEBUG nextflow.Session - Executor pool size: 10 Sep-26 13:21:24.988 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null Sep-26 13:21:24.995 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=30; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false Sep-26 13:21:25.019 [main] DEBUG nextflow.cli.CmdRun - Version: 24.04.4 build 5917 Created: 01-08-2024 07:05 UTC (09:05 CEST) System: Linux 3.10.0-1160.119.1.el7.x86_64 Runtime: Groovy 4.0.21 on Java HotSpot(TM) 64-Bit Server VM 11.0.9+7-LTS Encoding: UTF-8 (UTF-8) Process: @.*** [172.18.10.236] CPUs: 10 - Mem: 125.4 GB (68.8 GB) - Swap: 8 GB (7.2 GB) Sep-26 13:21:25.062 [main] DEBUG nextflow.Session - Work-dir: /crex/proj/naiss2024-23-81/private/vathsa/sc/PD/work [lustre] Sep-26 13:21:25.136 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[] Sep-26 13:21:25.150 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory Sep-26 13:21:25.278 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory Sep-26 13:21:25.302 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 11; maxThreads: 1000 Sep-26 13:21:25.587 [main] DEBUG nextflow.Session - Session start Sep-26 13:21:25.591 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /proj/naiss2024-23-81/private/vathsa/sc/PD/results/pipeline_info/execution_trace_2024-09-26_13-21-24.txt Sep-26 13:21:25.606 [main] DEBUG nextflow.Session - Using default localLib path: /proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home/assets/nf-core/scrnaseq/lib Sep-26 13:21:25.613 [main] DEBUG nextflow.Session - Adding to the classpath library: /proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home/assets/nf-core/scrnaseq/lib Sep-26 13:21:25.822 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution Sep-26 13:21:33.444 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [paramsHelp:paramsHelp]; plugin Id: nf-validation Sep-26 13:21:33.496 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [paramsSummaryLog:paramsSummaryLog]; plugin Id: nf-validation Sep-26 13:21:33.497 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [validateParameters:validateParameters]; plugin Id: nf-validation Sep-26 13:21:33.501 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [paramsSummaryMap:paramsSummaryMap]; plugin Id: nf-validation Sep-26 13:21:33.502 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [fromSamplesheet:fromSamplesheet]; plugin Id: nf-validation Sep-26 13:21:33.757 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [paramsSummaryLog:paramsSummaryLog, paramsSummaryMap:paramsSummaryMap]; plugin Id: nf-validation Sep-26 13:21:33.908 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [paramsHelp:paramsHelp]; plugin Id: nf-validation Sep-26 13:21:33.909 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [paramsSummaryLog:paramsSummaryLog]; plugin Id: nf-validation Sep-26 13:21:33.909 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [validateParameters:validateParameters]; plugin Id: nf-validation Sep-26 13:21:33.912 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [paramsSummaryMap:paramsSummaryMap]; plugin Id: nf-validation Sep-26 13:21:33.913 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [fromSamplesheet:fromSamplesheet]; plugin Id: nf-validation Sep-26 13:21:34.294 [main] DEBUG nextflow.Nextflow - Using schema file: nextflow_schema.json Sep-26 13:21:34.337 [main] INFO nextflow.Nextflow -

                                    ,--./,-.
    ___     __   __   __   ___     /,-._.--~'

|\ | | / / \ |__) |__ } { | \| | \__, \__/ | \ |___ \-.,--, .,._,' nf-core/scrnaseq v2.7.1-g4171377

Core Nextflow options revision : master runName : elegant_mayer containerEngine : singularity launchDir : /crex/proj/naiss2024-23-81/private/vathsa/sc/PD workDir : /crex/proj/naiss2024-23-81/private/vathsa/sc/PD/work projectDir : /proj/naiss2024-23-81/private/vathsa/sc/PD/nxf-home/assets/nf-core/scrnaseq userName : josag897 profile : uppmax configFiles :

Input/output options input : /proj/naiss2024-23-81/private/vathsa/sc/PD/scSample_sheet2.csv outdir : /proj/naiss2024-23-81/private/vathsa/sc/PD/results

Mandatory arguments aligner : cellranger protocol : 10XV2

Reference genome options fasta : /sw/data/iGenomes/Homo_sapiens/NCBI/GRCh38/Sequence/WholeGenomeFasta/genome.fa gtf : /sw/data/iGenomes/Homo_sapiens/NCBI/GRCh38/Annotation/Genes.gencode/genes.gtf save_reference : true igenomes_base : /sw/data/igenomes/

Institutional config options config_profile_description: UPPMAX (Rackham) cluster profile provided by nf-core/configs. config_profile_contact : Phil Ewels @.***) config_profile_url : https://www.uppmax.uu.se/

Max job request options max_cpus : 20 max_memory : 970 GB max_time : 10d

!! Only displaying parameters that differ from the pipeline defaults !!

If you use nf-core/scrnaseq for your analysis please cite:

grst commented 1 month ago

Hmm, that doesn't contain any error message at all (it seems truncated). Not sure if it was chopped off by github?

I was actually hoping to see the .command.log file of the process that failed. It's located in the work directory of that process, e.g. /crex/proj/naiss2024-23-81/private/vathsa/sc/PD/work/f2/71f5c948cbd4bce16286fbc597e43c/.command.log

JosephAgim commented 1 month ago

alright here it is. Thanks for the direction!

Den ons 2 okt. 2024 kl 11:42 skrev Gregor Sturm @.***>:

Hmm, that doesn't contain any error message at all (it seems truncated). Not sure if it was chopped off by github?

I was actually hoping to see the .command.log file of the process that failed. It's located in the work directory of that process, e.g.

/crex/proj/naiss2024-23-81/private/vathsa/sc/PD/work/f2/71f5c948cbd4bce16286fbc597e43c/.command.log

— Reply to this email directly, view it on GitHub https://github.com/nf-core/scrnaseq/issues/372#issuecomment-2388093155, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUGZAPUGUBEHSWQAFFGDYY3ZZO5ZXAVCNFSM6AAAAABO6X6AXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBYGA4TGMJVGU . You are receiving this because you authored the thread.Message ID: @.***>

INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred INFO: Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred INFO: Environment variable SINGULARITYENV_SNIC_TMP is set, but APPTAINERENV_SNIC_TMP is preferred

Martian Runtime - v4.0.12 Serving UI at http://r1226.uppmax.uu.se:38523?auth=MfInzL4Q0HSOQayAOcVJb8Y5oYINtAUoa-1wYc4STTU

Running preflight checks (please wait)... Checking sample info... Checking FASTQ folder... Checking reference... Checking reference_path (/crex/proj/naiss2024-23-81/private/vathsa/sc/PD/work/df/7a73831405f8b0efd037a128271cdc/cellranger_reference) on r1226.uppmax.uu.se... Checking optional arguments... mro: v4.0.12 mrp: v4.0.12 Anaconda: Python 3.10.11 numpy: 1.24.3 scipy: 1.10.1 pysam: 0.21.0 h5py: 3.6.0 pandas: 2.1.4 STAR: 2.7.2a samtools: samtools 1.16.1 2024-09-26 21:50:39 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.SANITIZE_MAP_CALLS 2024-09-26 21:50:39 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_CHEMISTRY_DETECTOR.CHECK_SINGLE_BEAM_MODE 2024-09-26 21:50:39 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.COUNT_GEM_WELL_PROCESSOR._BASIC_SC_RNA_COUNTER._MATRIX_COMPUTER.DISABLE_CORRECTION_FACTOR 2024-09-26 21:50:39 [runtime] (run:local) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.COUNT_GEM_WELL_PROCESSOR._BASIC_SC_RNA_COUNTER._MATRIX_COMPUTER.DISABLE_CORRECTION_FACTOR.fork0.chnk0.main 2024-09-26 21:50:39 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.WRITE_GENE_INDEX 2024-09-26 21:50:39 [runtime] (run:local) ID.SRR26129865_1.SC_RNA_COUNTER_CS.WRITE_GENE_INDEX.fork0.chnk0.main 2024-09-26 21:50:39 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.COPY_VDJ_REFERENCE 2024-09-26 21:50:39 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.COPY_CHEMISTRY_SPEC 2024-09-26 21:50:39 [runtime] (run:local) ID.SRR26129865_1.SC_RNA_COUNTER_CS.COPY_CHEMISTRY_SPEC.fork0.chnk0.main 2024-09-26 21:50:39 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.SPLIT_VDJ_INPUTS 2024-09-26 21:50:39 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.FULL_COUNT_INPUTS.WRITE_GENE_INDEX 2024-09-26 21:50:39 [runtime] (run:local) ID.SRR26129865_1.SC_RNA_COUNTER_CS.FULL_COUNT_INPUTS.WRITE_GENE_INDEX.fork0.chnk0.main 2024-09-26 21:50:39 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MAKE_FULL_CONFIG._MAKE_VDJ_CONFIG 2024-09-26 21:50:39 [runtime] (run:local) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MAKE_FULL_CONFIG._MAKE_VDJ_CONFIG.fork0.chnk0.main 2024-09-26 21:50:40 [runtime] (chunks_complete) ID.SRR26129865_1.SC_RNA_COUNTER_CS.COPY_CHEMISTRY_SPEC 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_CHEMISTRY_DETECTOR.DETECT_COUNT_CHEMISTRY 2024-09-26 21:50:40 [runtime] (run:local) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_CHEMISTRY_DETECTOR.DETECT_COUNT_CHEMISTRY.fork0.chnk0.main 2024-09-26 21:50:40 [runtime] (chunks_complete) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MAKE_FULL_CONFIG._MAKE_VDJ_CONFIG 2024-09-26 21:50:40 [runtime] (chunks_complete) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.COUNT_GEM_WELL_PROCESSOR._BASIC_SC_RNA_COUNTER._MATRIX_COMPUTER.DISABLE_CORRECTION_FACTOR 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GEM_WELL_PROCESSOR.SETUP_VDJ_DEMUX 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_B_GEM_WELL_PROCESSOR.SETUP_VDJ_DEMUX 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GD_GEM_WELL_PROCESSOR.SETUP_VDJ_DEMUX 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GEM_WELL_PROCESSOR.MULTI_SETUP_CHUNKS 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GD_GEM_WELL_PROCESSOR.MULTI_SETUP_CHUNKS 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GD_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.MAKE_SHARD 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.COUNT_GEM_WELL_PROCESSOR._BASIC_SC_RNA_COUNTER._MATRIX_COMPUTER.COMPUTE_CORRECTION_FACTOR 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_B_GEM_WELL_PROCESSOR.MULTI_SETUP_CHUNKS 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_B_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.MAKE_SHARD 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.MAKE_SHARD 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_B_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.BARCODE_CORRECTION 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GD_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.BARCODE_CORRECTION 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_B_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.RUST_BRIDGE 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.BARCODE_CORRECTION 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GD_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.RUST_BRIDGE 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_B_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.ASSEMBLE_VDJ 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GD_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.ASSEMBLE_VDJ 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.RUST_BRIDGE 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_B_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.MERGE_METRICS 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.ASSEMBLE_VDJ 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GD_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.MERGE_METRICS 2024-09-26 21:50:40 [runtime] (ready) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.VDJ_T_GEM_WELL_PROCESSOR.SC_VDJ_CONTIG_ASSEMBLER.MERGE_METRICS 2024-09-26 21:50:40 [runtime] (failed) ID.SRR26129865_1.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_CHEMISTRY_DETECTOR.DETECT_COUNT_CHEMISTRY

[error] Pipestance failed. Error log at: SRR26129865_1/SC_RNA_COUNTER_CS/SC_MULTI_CORE/MULTI_CHEMISTRY_DETECTOR/DETECT_COUNT_CHEMISTRY/fork0/chnk0-u004df5bb10/_errors

Log message: R1 and R2 reads identical in sample "SRR26129865_1" at "/scratch/50513097/nxf.vtwL0u2PTy/fastq_all"

Waiting 6 seconds for UI to do final refresh. Pipestance failed. Use --noexit option to keep UI running after failure.

2024-09-26 21:50:46 Shutting down. Saving pipestance info to "SRR26129865_1/SRR26129865_1.mri.tgz" For assistance, upload this file to 10x Genomics by running:

cellranger upload "SRR26129865_1/SRR26129865_1.mri.tgz"

Traceback (most recent call last): File "/crex/proj/naiss2024-23-81/private/vathsa/sc/PD/work/f2/71f5c948cbd4bce16286fbc597e43c/.command.sh", line 59, in run( File "/opt/conda/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['cellranger', 'count', '--id', 'SRR26129865_1', '--fastqs', 'fastq_all', '--transcriptome', 'cellranger_reference', '--localcores', '12', '--localmem', '72', '--chemistry', 'SC3Pv2', '--create-bam', 'true']' returned non-zero exit status 1.

grst commented 1 month ago

Ok, that reveals the actual error message: 

Log message:
R1 and R2 reads identical in sample "SRR26129865_1" at "/scratch/50513097/nxf.vtwL0u2PTy/fastq_all"

Can you please first verify that the files as specified in the samplesheet are really not identical? Maybe this was a mistake during download or renaming.

If they are correct, then please check the fastq_all folder in that process work directory that contains the files as they were renamed by nextflow. If the issue is only there, then it could indeed be a bug in the autorename feature.

JosephAgim commented 1 month ago

copyied from the samplesheet.csv: sample,fastq_1,fastq_2 SRR26129865_1,/proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R1_001.fastq.gz,/proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz copied from the fastq dirr: SRX21843315_SRR26129865_R1_001.fastq.gz SRX21843315_SRR26129865_R2_001.fastq.gz

Den ons 2 okt. 2024 kl 12:05 skrev Gregor Sturm @.***>:

Reopened #372 https://github.com/nf-core/scrnaseq/issues/372.

— Reply to this email directly, view it on GitHub https://github.com/nf-core/scrnaseq/issues/372#event-14486120196, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUGZAPQTBS3WFAAC34TAH7DZZPAN3AVCNFSM6AAAAABO6X6AXGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGQ4DMMJSGAYTSNQ . You are receiving this because you authored the thread.Message ID: @.***>

grst commented 1 month ago

yeah, the filenames are different, but if you look at the actual files (e.g. using less, or just calculating md5sums), are they any different?

JosephAgim commented 1 month ago

there is not fastq_all in any dirr only fastq and it is not in the work dirr

Den ons 2 okt. 2024 kl 12:57 skrev Joseph Agi Maqdissi < @.***>:

copyied from the samplesheet.csv: sample,fastq_1,fastq_2

SRR26129865_1,/proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R1_001.fastq.gz,/proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz copied from the fastq dirr: SRX21843315_SRR26129865_R1_001.fastq.gz SRX21843315_SRR26129865_R2_001.fastq.gz

Den ons 2 okt. 2024 kl 12:05 skrev Gregor Sturm @.***

:

Reopened #372 https://github.com/nf-core/scrnaseq/issues/372.

— Reply to this email directly, view it on GitHub https://github.com/nf-core/scrnaseq/issues/372#event-14486120196, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUGZAPQTBS3WFAAC34TAH7DZZPAN3AVCNFSM6AAAAABO6X6AXGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGQ4DMMJSGAYTSNQ . You are receiving this because you authored the thread.Message ID: @.***>

grst commented 1 month ago

hmm... but let's first check the input files. What's the result of 

md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R1_001.fastq.gz
md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz
JosephAgim commented 1 month ago

sorry for the late respons:

here is the input and output: @.*** fastq]$ md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz beda4ddc420e7afde11a1e4a698cd260 /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz

Den ons 2 okt. 2024 kl 13:14 skrev Gregor Sturm @.***>:

hmm... but let's first check the input files. What's the result of

md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R1_001.fastq.gz md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz

— Reply to this email directly, view it on GitHub https://github.com/nf-core/scrnaseq/issues/372#issuecomment-2388390019, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUGZAPR647ZOMVDPJBAQBYLZZPITFAVCNFSM6AAAAABO6X6AXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBYGM4TAMBRHE . You are receiving this because you authored the thread.Message ID: @.***>

JosephAgim commented 1 month ago

sorry this is the input and output: @. fastq]$ md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R1_001.fastq.gz beda4ddc420e7afde11a1e4a698cd260 /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R1_001.fastq.gz @. fastq]$ md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz beda4ddc420e7afde11a1e4a698cd260 /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz

Den ons 2 okt. 2024 kl 14:44 skrev Joseph Agi Maqdissi < @.***>:

sorry for the late respons:

here is the input and output: @.*** fastq]$ md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz beda4ddc420e7afde11a1e4a698cd260 /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz

Den ons 2 okt. 2024 kl 13:14 skrev Gregor Sturm @.***

:

hmm... but let's first check the input files. What's the result of

md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R1_001.fastq.gz md5sum /proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_R2_001.fastq.gz

— Reply to this email directly, view it on GitHub https://github.com/nf-core/scrnaseq/issues/372#issuecomment-2388390019, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUGZAPR647ZOMVDPJBAQBYLZZPITFAVCNFSM6AAAAABO6X6AXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBYGM4TAMBRHE . You are receiving this because you authored the thread.Message ID: @.***>

grst commented 1 month ago

ok, so SRX21843315_SRR26129865_R1_001.fastq.gz and SRX21843315_SRR26129865_R2_001.fastq.gz are the same files actually. Then it seems to be a problem with the input data rather than the pipeline.

JosephAgim commented 1 month ago

SRX21843315_SRR26129865_1.fastq.gz SRX21843315_SRR26129866_1.fastq.gz SRX21843315_SRR26129865_2.fastq.gz SRX21843315_SRR26129866_2.fastq.gz SRX21843315_SRR26129865_3.fastq.gz SRX21843315_SRR26129866_3.fastq.gz

The files above i got from nf-core fetching. this is the fastq file i fetched for : SRR26129866.

soo in the pipeline the for nf-core scRNAseq they expect you to do this: sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz

I have tried to have different files. Like you said one should not have to rename files: input from samplesheet.csv : sample,fastq_1,fastq_2 SRR26129865,/proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_1.fastq.gz,/proj/naiss2024-23-81/private/vathsa/sc/PD/fastq/SRX21843315_SRR26129865_2.fastq.gz

and i also tried with the replication number SRR26129865_1 at sample parameter just like the guide

with two of the files mentioned above ofc: SRX21843315_SRR26129865_1.fastq.gz SRX21843315_SRR26129865_2.fastq.gz

this did not work either I do not understand.

Den ons 2 okt. 2024 kl 15:20 skrev Gregor Sturm @.***>:

ok, so SRX21843315_SRR26129865_R1_001.fastq.gz and SRX21843315_SRR26129865_R2_001.fastq.gz are the same files actually. Then it seems to be a problem with the input data rather than the pipeline.

— Reply to this email directly, view it on GitHub https://github.com/nf-core/scrnaseq/issues/372#issuecomment-2388628884, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUGZAPSMD46PO4CA7C6EIJLZZPXJJAVCNFSM6AAAAABO6X6AXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBYGYZDQOBYGQ . You are receiving this because you authored the thread.Message ID: @.***>

grst commented 1 month ago

Sometimes SRA is weird and it doesn't properly return paired end reads as separate files. I had a similar case once and ended up contacting SRA support.

You could also try asking in #fetchngs on the nf-core slack if they have any insight or if it could be a bug in fetchngs.

Im pretty sure now though that the problem is not within scrnaseq, so I'm closing the issue here.

JosephAgim commented 1 month ago

Alright, even though I have redownloaded files. But i will make a post on slack fetchings and i will also check w the SRA support.

Thanks for the help

Best regards, Joseph

On Wed, 2 Oct 2024, 20:53 Gregor Sturm, @.***> wrote:

Sometimes SRA is weird and it doesn't properly return paired end reads as separate files. I had a similar case once and ended up contacting SRA support.

You could also try asking in #fetchngs on the nf-core slack if they have any insight or if it could be a bug in fetchngs.

Im pretty sure now though that the problem is not within scrnaseq, so I'm closing the issue here.

— Reply to this email directly, view it on GitHub https://github.com/nf-core/scrnaseq/issues/372#issuecomment-2389462898, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUGZAPWSO2LRY6NKM3KCMWTZZQ6MBAVCNFSM6AAAAABO6X6AXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBZGQ3DEOBZHA . You are receiving this because you authored the thread.Message ID: @.***>