Cell Ranger ATAC Count Fails with gzip.BadGzipFile

Hello Cell Ranger team,

I am currently running cellranger-atac count using a SLURM script on a remote server. However, the pipeline crashes during the ATAC_SETUP_CHUNKS stage with the error gzip.BadGzipFile: Not a gzipped file.

The issue appears in the log file and points to an invalid gzip header. I suspect it may be related to the FASTQ files or disk space limitations.

This is the script:

#!/bin/bash
#SBATCH --job-name=cellranger
#SBATCH --time=7-00:00:00
#SBATCH --mem=256gb
#SBATCH -e cellranger_atac.err
#SBATCH -o cellranger_atac.out
#SBATCH --partition=scu-cpu
export PATH=/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0:$PATH

cellranger-atac count --id=HPAP-035 \
   --fastqs=/athena/cayuga_0026/scratch/sel4008/t1d_multiome/hpapdata/HPAP-035/fastq \
   --sample=HPAP-035_FGC1984_6 \
   --reference=/athena/cayuga_0026/scratch/sel4008/t1d_multiome/refdata-cellranger-arc-GRCh38-2020-A-2.0.0

This is what I see in the cellranger_atac.out file.

Martian Runtime - v4.0.7
2024-11-15 17:15:01 [runtime] Reattaching in local mode.
Serving UI at http://c0002:38311?auth=LOh4cxh2n1u0uihwwlidKb2aEZ9v2O6qZobmfSaCbBw

2024-11-15 17:15:01 [runtime] (reset-partial)   ID.HPAP-035.SC_ATAC_COUNTER_CS.SC_ATAC_COUNTER._BASIC_SC_ATAC_COUNTER.ATAC_SETUP_CHUNKS.fork0.chnk0
2024-11-15 17:15:01 [runtime] Found orphaned local stage: ID.HPAP-035.SC_ATAC_COUNTER_CS.SC_ATAC_COUNTER._BASIC_SC_ATAC_COUNTER.ATAC_SETUP_CHUNKS
2024-11-15 17:15:01 [runtime] (split_complete)  ID.HPAP-035.SC_ATAC_COUNTER_CS.SC_ATAC_COUNTER._BASIC_SC_ATAC_COUNTER.ATAC_SETUP_CHUNKS
2024-11-15 17:15:01 [runtime] (run:local)       ID.HPAP-035.SC_ATAC_COUNTER_CS.SC_ATAC_COUNTER._BASIC_SC_ATAC_COUNTER.ATAC_SETUP_CHUNKS.fork0.chnk0.main
2024-11-15 17:15:03 [runtime] (failed)          ID.HPAP-035.SC_ATAC_COUNTER_CS.SC_ATAC_COUNTER._BASIC_SC_ATAC_COUNTER.ATAC_SETUP_CHUNKS

[error] Pipestance failed. Error log at:
HPAP-035/SC_ATAC_COUNTER_CS/SC_ATAC_COUNTER/_BASIC_SC_ATAC_COUNTER/ATAC_SETUP_CHUNKS/fork0/chnk0-u538e37c7e5/_errors

Log message:
Traceback (most recent call last):
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/external/martian/adapters/python/martian_shell.py", line 680, in _main
    stage.main()
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/external/martian/adapters/python/martian_shell.py", line 639, in main
    self._run(lambda: self._module.main(args, outs))
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/external/martian/adapters/python/martian_shell.py", line 596, in _run
    cmd()
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/external/martian/adapters/python/martian_shell.py", line 639, in <lambda>
    self._run(lambda: self._module.main(args, outs))
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/mro/atac/stages/processing/setup_chunks/__init__.py", line 188, in main
    (flowcell, lane) = get_run_data(r1)
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/mro/atac/stages/processing/setup_chunks/__init__.py", line 281, in get_run_data
    name = next(gen)[0]
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/lib/python/tenkit/fasta.py", line 211, in read_generator_fastq
    yield read()
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/lib/python/tenkit/fasta.py", line 167, in read
    line1 = six.ensure_binary(next(line).strip())  # type: bytes
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/external/anaconda/lib/python3.8/gzip.py", line 390, in readline
    return self._buffer.readline(size)
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/external/anaconda/lib/python3.8/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/external/anaconda/lib/python3.8/gzip.py", line 479, in read
    if not self._read_gzip_header():
  File "/athena/cayuga_0026/scratch/sel4008/t1d_multiome/cellranger-atac-2.1.0/external/anaconda/lib/python3.8/gzip.py", line 427, in _read_gzip_header
    raise BadGzipFile('Not a gzipped file (%r)' % magic)
gzip.BadGzipFile: Not a gzipped file (b'<h')

Waiting 6 seconds for UI to do final refresh.
Pipestance failed. Use --noexit option to keep UI running after failure.

2024-11-15 17:15:09 Shutting down.

Could you please provide guidance on how to troubleshoot this issue? Thank you for your help!

10XGenomics / cellranger

Cell Ranger ATAC Count Fails with gzip.BadGzipFile #253