Failure to fetch Singularity image

[jainronit]

Operating System

macOS

Other Linux

No response

Workflow Version

v23.04.3

Workflow Execution

Command line (Cluster)

Other workflow execution

No response

EPI2ME Version

No response

CLI command run

nextflow run epi2me-labs/wf-transcriptomes --de_analysis --direct_rna --fastq differential_expression/reads/ --minimum_mapping_quality 20 --ref_annotation differential_expression/GRCh38.gtf --ref_genome differential_expression/GRCh38.fa --sample_sheet differential_expression/sample_sheet.csv -profile singularity

Workflow Execution - CLI Execution Profile

singularity

What happened?

When trying to run the workflow through Singularity, I got an error where the image could not be fetched since there was no descriptor found for the reference (see log output below). I also looked for a log file output from Nextflow but it seems that there was none generated in the directory from which I ran the command. I'd appreciate any help troubleshooting!

Relevant log output

Error executing process > 'validate_sample_sheet'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name ontresearch-wf-common-sha8b5843d549bb210558cbb676fe537a153ce771d6.img.pulling.1721430658472 docker://ontresearch/wf-common:sha8b5843d549bb210558cbb676fe537a153ce771d6 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:48a28224450f92e055b1bfb8f5ce40d91488bd4eb8e5143872077a6bbdb66805
    Copying blob sha256:9ea8908f47652b59b8055316d9c0e16b365e2b5cee15d3efcb79e2957e3e7cad
    Copying blob sha256:249f61f0164bcbfa4cc9eec0fe362885248521a49bbdb32c99bc45ecd813f7a4
    Copying blob sha256:48a28224450f92e055b1bfb8f5ce40d91488bd4eb8e5143872077a6bbdb66805
    Copying blob sha256:a8fe4e569532cc444b897610796cc284c98caa9849201bfc1dbb8510f7ca3727
    Copying blob sha256:2fb1b4a299c1b9e2edc1b6a92612b13f5c8ab254defef223c271ef88e1bba6c8
    Copying blob sha256:ebd1d27baee5b5df9794803af4b951a629364816534c859839fde66d1ae4b68c
    Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
    Copying blob sha256:927c8bfa8a9da7025a6a701bcabf8410a0bb10ff1cfc47c0ecc1cfacd75e274d
    Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
    Copying blob sha256:68f3509bb7ca242ab9270241cc40ffd5bb43e711c62a137e6fa74fa346f0576c
    Copying blob sha256:f1e31f0953f979bedae5ca7248b686b4ed7c0af2474b56586b71b09b984ff3fe
    Copying blob sha256:a0ae10c439e3d6d12d419420151c2972a086874799c166822348dc54c04bfaa2
    Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
    Copying blob sha256:cc0bd25cbf8a647cae9d4c38edd677f04263811334e69986b39d3e8cd49c4d1b
    Copying blob sha256:73f71347d6dda2fac6b2f4eca36b090d1839ba5d5976664ee53d2aa2ab3d5280
    Copying blob sha256:d0ca1333c3d5079c767569ba2681afa193c36a5bcca9979504f88683c2aeb607
    Copying blob sha256:ca2356b1c42ae1d4cd3ddcf9de409034f73d612a0981d412099629086632d595
    Copying blob sha256:f7f93d1d1b6a059c1fbe6857071a5329c719c737b88a2ea45f46906b0c1e2730
    Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
    Copying config sha256:c8ad914682de8c371f83291ce4837e044728398e556439ec4ab8563c6254112c
    Writing manifest to image destination
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: while getting config: no descriptor found for reference "75c8bc4a88c25a7790759a5e76d532c8081aaaa2b894191c6720ace2b77ecce4"

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

yes

Other demo data information

No response

[sarahjeeeze]

Hi, can you successfully run the workflow with the demo data provided with singularity?

[jainronit]

Hi, no the demo data does not work with singularity. I get the following error:

FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: while creating squashfs: create command failed

[sarahjeeeze]

And can you normally run a programme such as hello world with singularity on your cluster?? It looks like it could be an issue with your singularity set up and you might want to contact your cluster administrator.

[jainronit]

Hi @sarahjeeeze, I managed to successfully run the workflow with the demo data with Singularity. However, when running the workflow on my actual samples, I am running into an issue where there seems to be a problem with parsing a bam file.

The full output is attached here for your reference: slurm-50705236.txt

And here is the relevant error message:

ERROR ~ Error executing process > 'pipeline:differential_expression:map_transcriptome (2)'

Caused by: Process pipeline:differential_expression:map_transcriptome (2) terminated with an error exit status (1)

Command executed:

minimap2 -t 4 -ax splice -uf -p 1.0 "genome_index.mmi" "seqs.fastq.gz" | samtools view -Sb > "output.bam" samtools sort -@ 4 "output.bam" -o "PUS7KDrep2_reads_aln_sorted.bam"

Command exit status: 1

Command output: (empty)

Command error: INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred [WARNING] Indexing parameters (-k, -w or -H) overridden by parameters used in the prebuilt index. [M::main::27.5740.23] loaded/built the index for 130702 target sequence(s) [M::mm_mapopt_update::28.5940.26] mid_occ = 3254 [M::mm_idx_stat] kmer size: 14; skip: 10; is_hpc: 0; #seq: 130702 [M::mm_idx_stat::28.937*0.27] distinct minimizers: 35748550 (8.85% are singletons); average occurrences: 30.060; average spacing: 5.357; total length: 5756717400 [E::sam_hdr_create] Invalid header line: must start with @HD/@SQ/@RG/@PG/@CO [main_samview] fail to read the header from "-".

[sarahjeeeze]

To help me recreate your error - where did you get the reference genome and annotation files from? Could you point me at the exact files? Also your reads are definitely --direct_rna not cdna?

[jainronit]

I got the reference genome and annotation files from GENCODE (https://www.gencodegenes.org/human/release_46.html). Specifically, I am using the GRCh38.p14 genome assembly and the basic gene annotation gtf file. And yes, my reads are direct RNA, not cDNA.

Here is the exact command I am running:

nextflow run epi2me-labs/wf-transcriptomes \ --de_analysis \ --direct_rna \ --fastq 'differential_expression/reads/' \ --ref_annotation 'differential_expression/GRCh38.gtf' \ --minimum_mapping_quality 20 \ --ref_genome 'differential_expression/GRCh38.fa' \ --sample_sheet 'differential_expression/sample_sheet.csv' \ -profile singularity

[sarahjeeeze]

hmm inputs seem fine - To rule out if being a memory error could you add this to the nextflow.config - at the end of the file

process {
    withName: 'map_transcriptome' {
        memory = 32.GB
        }
}

[jainronit]

Hi @sarahjeeeze, I am still getting the same error so it doesn't seem like it's a memory error? Could you let me know if there's any test I can run to see why samtools view seems to be having issues reading the header of the sam file? Is there a particular file that I should go look at to see if it might be malformed? I could also send you the input data to see if you can recreate the error on your end.

[sarahjeeeze]

hi, to your input cmd can you try adding --minimap2_index_opts '-k 15', if that doesnt work please do share your data or a subsample with me as i am struggling to recreate your error with my reads and your input references

for ref_genome are you using Genome sequence (GRCh38.p14) - GRCh38.p14.genome.fa.gz

[jainronit]

Hi Sarah, just to update you, I ended up resolving this issue by increasing the memory allocated to minimap2 to 48 GB in the nextflow.config file. Thank you so much for all your help, I really appreciate it!

[sarahjeeeze]

Great thanks, we are working on some memory improvements for minimap steps.