alfpark
commented
5 years ago You have two issues here as far as I can tell:
- The compute node preparation task is failing when mounting your Azure File Share. Please ensure that the file share named
fileshareexists in storage accountagcanrnaseqdiag. When the compute node mounts the specified file share, it must exist in Azure Storage (as per documentation). - Your Docker image reference is incorrect. The Docker image must be fully qualified as per documentation. As-is in your
config.yamlandjobs.yamlfile, the image pull attempt would happen against Docker Hub, certainly not what you want in your case. You need to modify the image name in both yaml files to be:agcanregistry.azurecr.io/trimmomatic/trimmomatic.
markpearl
I have come across an issue where my tasks submitted using the Batch Shipyard command line tools are stuck in the Preparing state because the Docker image they depend on is not present on the Batch compute node.
After dealing with the issue for quite some time, I created an admin user on the compute node and logged in using ssh. I manually downloaded my Docker image, which is placed in a private registry, and then ran my task again. This time the task failed again and is hanging.
Provided are my configuration files:
config.yaml
Dockerfile used to create the image:
FROM biocontainers/biocontainers:latest
RUN conda install trimmomatic
WORKDIR /home/mjpearl/fileshare
CMD ["trimmomatic"]
This image is stored in my container registry under the trimmomatic repo and trimmomatic for the tag (i.e. trimmomatic: trimmomatic ).
As per azure-hpc example, my snakemake workflow file is creating a shell script with the following command:
mjpearl@controlvm:~/fileshare/snakemake$ $SHIPYARD/shipyard pool add --configdir $FILESHARE/snakemake/azurebatch
The pool get's created successfully, and as soon as I see that the pool reaches "starting" stats i Abort the command.
When I run the snakemake workflow the job just hangs:
For the batch account, the start task for the dedicated node seems to fail:
Also provided are the stderr.txt and stdout.txt, respectively:
Stderr.txt: mount error(2): No such file or directory Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Stdout.txt: Linux 7bdca5853a074102b5577ae7ba27e2e1000000 4.18.0-1013-azure #13~18.04.1-Ubuntu SMP Thu Feb 28 23:48:47 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux 2019-03-18T04:51:49,180846313+00:00 - INFO - Prep start
Configuration: Custom image: 0 Native mode: 0 OS Distribution: ubuntu 18.04 Batch Shipyard version: 3.7.0 Blobxfer version: 1.7.0 Singularity version: User mountpoint: /mnt Mount path: /mnt/batch/tasks/mounts Batch Insights: 0 Prometheus: NE=, CA=, Network optimization: 0 Encryption cert thumbprint: Install Kata Containers: 0 Default container runtime: runc Install BeeGFS BeeOND: 0 Storage cluster mount: Custom mount: Install LIS: GPU: Azure Blob: 0 Azure File: 1 GlusterFS on compute: 0 HPN-SSH: 0 Enable Azure Batch group for Docker access: Fallback registry: Docker image preload delay: 0 Cascade via container: 1 P2P: 0 Block on images: trimmomatic/trimmomatic#
2019-03-18T04:51:49,296618240+00:00 - INFO - LIS installation not required 2019-03-18T04:51:49,305637489+00:00 - INFO - Mounting Azure File
Provided is the snakemake file:
Snakefile for the RNA-Seq analysis pipeline using test data from zebrafish
You should not need to edit this file unless you are changing the programs in the pipeline
configfile: "config_zebrafish.yaml" SAMPLES = config['samples']
R1_suffix=config['input_file_R1_suffix'] R2_suffix=config['input_file_R2_suffix'] genome_fasta_file = config['genome_fasta_file'] genome_index_base = config['genome_index_base'] merged_transcripts_file=config['merged_transcripts_file']
rule all: input: abundance_table=expand("4_transcript_abundances/{sample}/{sample}.abundance_table.txt", sample=SAMPLES)
rule trim_and_qc_all: input: html=expand("1_fastqc_reports/{sample}_R1.trimmed_paired_fastqc.html", sample=SAMPLES)
rule assemble_all: input: gtf=expand("3_assembled_transcripts/{sample}.gtf", sample=SAMPLES)
rule map_all: input: bam=expand("2_mapped_reads/{sample}.sorted.bam", sample=SAMPLES)
rule trim_reads: input: R1_reads="/home/mjpearl/fileshare/data/{sample}" + R1_suffix, R2_reads="/home/mjpearl/fileshare/data/{sample}" + R2_suffix
output: "1_trimmed_reads/{sample}_R1.trimmed_paired.fastq", "1_trimmed_reads/{sample}_R1.trimmed_unpaired.fastq", "1_trimmed_reads/{sample}_R2.trimmed_paired.fastq", "1_trimmed_reads/{sample}_R2.trimmed_unpaired.fastq"
threads: config['threads'] params: run_params=config['trimmomatic_params'] shell: "echo -e \"#!/usr/bin/env bash\ncd $FILESHARE/snakemake;\n trimmomatic PE -threads {threads} {input.R1_reads} {input.R2_reads} {output} {params.run_params}\" > $FILESHARE/snakemake/jobrun.sh ;\n $SHIPYARD/shipyard jobs add --configdir $FILESHARE/snakemake/azurebatch --tail stderr.txt\n"
jobrun.sh get's created in the snakemake directory and created the following command:
trimmomatic PE -threads 1 /home/mjpearl/fileshare/data/zebrafish_6h_1.fastq /home/mjpearl/fileshare/data/zebrafish_6h_2.fastq 1_trimmed_reads/zebrafish_6h_R1.trimmed_paired.fastq 1_trimmed_reads/zebrafish_6h_R1.trimmed_unpaired.fastq 1_trimmed_reads/zebrafish_6h_R2.trimmed_paired.fastq 1_trimmed_reads/zebrafish_6h_R2.trimmed_unpaired.fastq HEADCROP:10 SLIDINGWINDOW:4:20 MINLEN:36
Any help would be appreciated.