Troubleshooting Dockerfile for DeepVariant on Mac M1: Issues with Bazel

[ghost]

I’m new to working with computers tools like DeepVariant. I’m trying to build DeepVariant using Docker on a Mac M1 and am encountering issues with the Dockerfile during the Bazel build process. I want to ensure compatibility with ARM64 architecture.

Docker version: Docker version 27.1.1, build 6312585 Bazel Version: 7.3.1 MacBook Model: M1 chip (ARM64 architecture)

Error:

My Dockerfilee code:

# Base image suitable for ARM64 architecture
FROM arm64v8/ubuntu:latest AS base

# Prevent interactive prompts
ENV DEBIAN_FRONTEND=noninteractive

# Install necessary packages
RUN apt-get update && \
    apt-get install -y \
    git \
    curl \
    unzip \
    wget \
    openjdk-17-jdk \
    build-essential \
    bzip2 \
    python3-pip \
    parallel && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Install Bazel (adjust version as needed)
RUN curl -LO "https://github.com/bazelbuild/bazel/releases/download/7.3.1/bazel-7.3.1-linux-arm64" && \
    chmod +x bazel-7.3.1-linux-arm64 && \
    mv bazel-7.3.1-linux-arm64 /usr/local/bin/bazel

# Install Conda
RUN curl -LO "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh" && \
    bash Miniconda3-latest-Linux-aarch64.sh -b -p /opt/miniconda && \
    rm Miniconda3-latest-Linux-aarch64.sh

# Setup Conda environment
ENV PATH="/opt/miniconda/bin:${PATH}"

RUN conda config --add channels defaults && \
    conda config --add channels bioconda && \
    conda config --add channels conda-forge && \
    conda create -n bio bioconda::bcftools bioconda::samtools -y && \
    conda clean -a

# Clone DeepVariant and build
FROM base AS builder

# Clone the DeepVariant repository
RUN git clone https://github.com/google/deepvariant.git /opt/deepvariant && \
    cd /opt/deepvariant && \
    git checkout tags/v1.6.1

# Run Bazel build with additional flags to skip problematic configurations
RUN bazel build -c opt --noincremental --experimental_action_listener= //deepvariant:make_examples //deepvariant:call_variants //deepvariant:postprocess_variants || { \
    echo "Bazel build failed"; \
    exit 1; }

# Final image
FROM base AS final

# Set environment variables
ENV VERSION=1.6.0
ENV PYTHON_VERSION=3.8
ENV PATH="/opt/miniconda/bin:${PATH}"

# Install Python packages
RUN pip install --upgrade pip setuptools wheel --timeout=120 && \
    pip install jaxlib jax --timeout=120 --extra-index-url https://storage.googleapis.com/jax-releases/jax_releases.html

# Copy DeepVariant binaries from the builder stage
COPY --from=builder /opt/deepvariant /opt/deepvariant
WORKDIR /opt/deepvariant

# Ensure executable scripts are correctly set up
RUN BASH_HEADER='#!/bin/bash' && \
    for script in make_examples call_variants call_variants_slim postprocess_variants vcf_stats_report show_examples runtime_by_region_vis multisample_make_examples labeled_examples_to_vcf make_examples_somatic train run_deepvariant run_deepsomatic; do \
        printf "%s\n%s\n" "${BASH_HEADER}" "python3 /opt/deepvariant/bin/${script}.zip \"$@\"" > /opt/deepvariant/bin/${script} && \
        chmod +x /opt/deepvariant/bin/${script}; \
    done

# Copy licenses and other necessary files
# Ensure these paths and URLs are correct and accessible
# Replace with valid URLs or remove if not needed
ADD https://storage.googleapis.com/deepvariant/models/DeepVariant/1.6.0/savedmodels/deepvariant.hybrid.savedmodel/saved_model.pb /models/
WORKDIR /opt/deepvariant/bin/
COPY --from=builder /opt/conda /opt/conda
COPY --from=builder /opt/deepvariant/run-prereq.sh .
COPY --from=builder /opt/deepvariant/settings.sh .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/make_examples.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/call_variants.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/call_variants_slim.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/postprocess_variants.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/vcf_stats_report.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/show_examples.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/runtime_by_region_vis.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/multisample_make_examples.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/labeler/labeled_examples_to_vcf.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/make_examples_somatic.zip  .
COPY --from=builder /opt/deepvariant/bazel-out/k8-opt/bin/deepvariant/train.zip  .
COPY --from=builder /opt/deepvariant/scripts/run_deepvariant.py .
COPY --from=builder /opt/deepvariant/scripts/run_deepsomatic.py .

RUN ./run-prereq.sh

RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 0 && \
    update-alternatives --install /usr/bin/python python /usr/bin/python${PYTHON_VERSION} 0

# Create shell wrappers for python zip files for easier use.
RUN \
  BASH_HEADER='#!/bin/bash' && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 /opt/deepvariant/bin/make_examples.zip "$@"' > \
    /opt/deepvariant/bin/make_examples && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 /opt/deepvariant/bin/call_variants.zip "$@"' > \
    /opt/deepvariant/bin/call_variants && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 /opt/deepvariant/bin/call_variants_slim.zip "$@"' > \
    /opt/deepvariant/bin/call_variants_slim && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 /opt/deepvariant/bin/postprocess_variants.zip "$@"' > \
    /opt/deepvariant/bin/postprocess_variants && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 /opt/deepvariant/bin/vcf_stats_report.zip "$@"' > \
    /opt/deepvariant/bin/vcf_stats_report && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 /opt/deepvariant/bin/show_examples.zip "$@"' > \
    /opt/deepvariant/bin/show_examples && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 /opt/deepvariant/bin/runtime_by_region_vis.zip "$@"' > \
    /opt/deepvariant/bin/runtime_by_region_vis && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 /opt/deepvariant/bin/multisample_make_examples.zip "$@"' > \
    /opt/deepvariant/bin/multisample_make_examples && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 -u /opt/deepvariant/bin/labeled_examples_to_vcf.zip "$@"' > \
    /opt/deepvariant/bin/labeled_examples_to_vcf && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 -u /opt/deepvariant/bin/make_examples_somatic.zip "$@"' > \
    /opt/deepvariant/bin/make_examples_somatic && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 -u /opt/deepvariant/bin/run_deepvariant.py "$@"' > \
    /opt/deepvariant/bin/run_deepvariant && \
  printf "%s\n%s\n" \
    "${BASH_HEADER}" \
    'python3 -u /opt/deepvariant/bin/run_deepsomatic.py "$@"' > \
    /opt/deepvariant/bin/run_deepsomatic

[kishwarshafin]

@mallorygw ,

If you are using docker, the best idea is to pull an existing version (pre-built) and use that. You should be able to do that by running:

docker pull google/deepvariant:1.6.1

[kishwarshafin]

I am going to close this issue for now. Please reopen if you need more help.