Can't install ANTs using Neurodocker due to Dropbox link error 403

[astewartau]

I tried to install ANTs v2.3.4 in my neurodocker recipe, and got this in my output. It points to the address https://dl.dropbox.com/s/gwf51ykkk5bifyj/ants-Linux-centos6_x86_64-v2.3.4.tar.gz which no longer exists.

#17 [10/27] RUN apt-get update -qq     && apt-get install -y -q --no-install-recommends            ca-certificates            curl            unzip     && rm -rf /var/lib/apt/lists/*     && echo "Downloading ANTs ..."     && mkdir -p /opt/ants-2.3.4     && curl -fsSL https://dl.dropbox.com/s/gwf51ykkk5bifyj/ants-Linux-centos6_x86_64-v2.3.4.tar.gz     | tar -xz -C /opt/ants-2.3.4 --strip-components 1
#17 11.60 Reading package lists...
#17 13.73 Building dependency tree...
#17 14.21 Reading state information...
#17 14.74 ca-certificates is already the newest version (20230311ubuntu0.18.04.1).
#17 14.74 curl is already the newest version (7.58.0-2ubuntu3.24).
#17 14.74 unzip is already the newest version (6.0-21ubuntu1.2).
#17 14.74 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
#17 14.76 Downloading ANTs ...
#17 15.37 curl: (22) The requested URL returned error: 403 
#17 15.38 
#17 15.38 gzip: stdin: unexpected end of file
#17 15.38 tar: Child returned status 1
#17 15.38 tar: Error is not recoverable: exiting now
#17 ERROR: process "/bin/sh -c apt-get update -qq     && apt-get install -y -q --no-install-recommends            ca-certificates            curl            unzip     && rm -rf /var/lib/apt/lists/*     && echo \"Downloading ANTs ...\"     && mkdir -p /opt/ants-2.3.4     && curl -fsSL https://dl.dropbox.com/s/gwf51ykkk5bifyj/ants-Linux-centos6_x86_64-v2.3.4.tar.gz     | tar -xz -C /opt/ants-2.3.4 --strip-components 1" did not complete successfully: exit code: 2
------
 > importing cache manifest from ghcr.io/***/qsmxt_7.0.0:
------
------
 > [10/27] RUN apt-get update -qq     && apt-get install -y -q --no-install-recommends            ca-certificates            curl            unzip     && rm -rf /var/lib/apt/lists/*     && echo "Downloading ANTs ..."     && mkdir -p /opt/ants-2.3.4     && curl -fsSL https://dl.dropbox.com/s/gwf51ykkk5bifyj/ants-Linux-centos6_x86_64-v2.3.4.tar.gz     | tar -xz -C /opt/ants-2.3.4 --strip-components 1:
#17 14.74 ca-certificates is already the newest version (20230311ubuntu0.18.04.1).
#17 14.74 curl is already the newest version (7.58.0-2ubuntu3.24).
#17 14.74 unzip is already the newest version (6.0-21ubuntu1.2).
#17 14.74 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
#17 14.76 Downloading ANTs ...
#17 15.37 curl: (22) The requested URL returned error: 403 
#17 15.38 
#17 15.38 gzip: stdin: unexpected end of file
#17 15.38 tar: Child returned status 1
#17 15.38 tar: Error is not recoverable: exiting now
------
qsmxt_7.0.0.Dockerfile:80
--------------------
  79 |         PATH="/opt/ants-2.3.4:$PATH"
  80 | >>> RUN apt-get update -qq \
  81 | >>>     && apt-get install -y -q --no-install-recommends \
  82 | >>>            ca-certificates \
  83 | >>>            curl \
  84 | >>>            unzip \
  85 | >>>     && rm -rf /var/lib/apt/lists/* \
  86 | >>>     && echo "Downloading ANTs ..." \
  87 | >>>     && mkdir -p /opt/ants-2.3.4 \
  88 | >>>     && curl -fsSL https://dl.dropbox.com/s/gwf51ykkk5bifyj/ants-Linux-centos6_x86_64-v2.3.4.tar.gz \
  89 | >>>     | tar -xz -C /opt/ants-2.3.4 --strip-components 1
  90 |     ENV PATH="/opt/dcm2niix-v1.0.20240202/bin:$PATH"
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update -qq     && apt-get install -y -q --no-install-recommends            ca-certificates            curl            unzip     && rm -rf /var/lib/apt/lists/*     && echo \"Downloading ANTs ...\"     && mkdir -p /opt/ants-2.3.4     && curl -fsSL https://dl.dropbox.com/s/gwf51ykkk5bifyj/ants-Linux-centos6_x86_64-v2.3.4.tar.gz     | tar -xz -C /opt/ants-2.3.4 --strip-components 1" did not complete successfully: exit code: 2
Error: Process completed with exit code 1.

[Remi-Gau]

any chance you can try building your recipe with an "apt" distro rather than a "yum" one.

Installing ANTs from source is generally broken and installing binaries usually fails on fedora and centos distros:

https://github.com/ReproNim/neurodocker/blob/test_docker_build/docs/README.md#ants

The image of the status of the test run may take a while to run.

https://camo.githubusercontent.com/6062afc473ee0bc2a87ef7b643c9360ba21ffc042d6ffe8f1ef70a57a867711c/687474703a2f2f6769746875622d616374696f6e732e3430616e74732e636f6d2f526570726f4e696d2f6e6575726f646f636b65722f6d61747269782e7376673f6272616e63683d746573745f646f636b65725f6275696c64266f6e6c793d616e7473

[astewartau]

This is the full recipe if that helps. It has package manager set to apt already?

#!/usr/bin/env bash
set -e

export toolName='qsmxt'
export toolVersion='7.0.0'

if [ "$1" != "" ]; then
    echo "Entering Debug mode"
    export debug=$1
fi

source ../main_setup.sh

# Version history:
# --qsmxtbase------------------------------------------------------------------------------------------------
# - 1.0.0: Release for preprint including FS7, Minc, FSL
# - 1.1.0: Change to Fastsurfer and replace minc with Ants
# - 1.1.1: update Julia and move packages to depot_Path + changed python version to 3.7
# - 1.1.2: changed python version back to 3.6 with older Miniconda version + fixed versions
# - 1.1.3: Removed FSL, bidscoin, matplotlib, seaborn; added bet2
# - 1.1.3 (fix): Fixed dependency problems occurring with Nipype (see github.com/QSMxT/QSMxT/runs/7553737387)
# --qsmxt----------------------------------------------------------------------------------------------------
# - 1.1.12: Combined qsmxt and qsmxtbase containers
# - 1.1.13: https://github.com/QSMxT/QSMxT/releases/tag/v1.1.13
# - 1.1.13 (container update): Added RomeoApp to Julia for nextQSM testing; removed run_1_fixGEphaseFFTshift.py from DEPLOY_BINS
# - 1.1.13 (container update): Added NeXtQSM
# - 1.2.0: Major update; added QSM.jl; ROMEO unwrapping; Laplacian unwrapping; V-SHARP; RTS QSM; major pipeline refactor
# - 1.3.0: Major update; added premade pipelines, interactive editor, PDF, TV, and fixed networkx version incompatibility
# - 1.3.0 (container update): Fixed FastSurfer to v1.1.1 due to seeming slowness in v2
# - ...
# - 3.2.0: Added fix for scikit-sparse due to Cython bug https://github.com/scikit-sparse/scikit-sparse/releases/tag/v0.4.9
# - 6.3.2: Note that Julia v1.10 is not compatible with QSM.jl - created issue https://github.com/kamesy/QSM.jl/issues/8

neurodocker generate ${neurodocker_buildMode} \
   --base-image ubuntu:18.04 \
   --pkg-manager apt \
   --env DEBIAN_FRONTEND=noninteractive \
   --run="printf '#!/bin/bash\nls -la' > /usr/bin/ll" \
   --run="chmod +x /usr/bin/ll" \
   --install bzip2 ca-certificates wget unzip gcc cmake g++ dbus-x11 libgtk2.0-0 git graphviz wget \
      zip libgl1 libglib2.0 libglu1-mesa libsm6 libxrender1 libxt6 libxcomposite1 libfreetype6 \
      libasound2 libfontconfig1 libxkbcommon0 libxcursor1 libxi6 libxrandr2 libxtst6 qt5-default \
      libqt5svg5-dev wget libqt5opengl5-dev libqt5opengl5 libqt5gui5 libqt5core5a libsuitesparse-dev \
      libsqlite3-dev \
   --env PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" \
   --run="mkdir -p ${mountPointList}" \
   --workdir="/opt/bet2" \
   --run="git clone https://github.com/liangfu/bet2.git . \
       && cmake . && make \
       && ln -s /opt/bet2/bin/bet2 /bin/bet" \
   --workdir="/opt" \
   --env SUBJECTS_DIR="/tmp" \
   --ants version=2.3.4 \
   --dcm2niix method=source version=v1.0.20240202 \
   --miniconda version=4.7.12.1 conda_install='python=3.8' \
   --run="rm -rf /usr/bin/python3.8 \
       && ln -s /opt/miniconda-latest/bin/python /usr/bin/python3.8 \
       && pip install setuptools==69.5.1 \
       && pip install qsmxt==${toolVersion} \
       && pip install dunamai \
       && pip install git+https://github.com/astewartau/nii2dcm.git@qsm \
       && nextqsm --download_weights" \
   --env PATH="\${PATH}:/opt/miniconda-latest/bin" \
   --run="git clone --depth 1 --branch v1.1.1 https://github.com/Deep-MI/FastSurfer.git /opt/FastSurfer \
       && sed -i 's/cu113/cpu/g' /opt/FastSurfer/requirements.txt \
       && pip install -r /opt/FastSurfer/requirements.txt" \
   --env FASTSURFER_HOME="/opt/FastSurfer" \
   --env PATH="\${PATH}:/opt/FastSurfer" \
   --copy test.sh /test.sh \
   --workdir="/opt/bru2" \
   --run="wget https://github.com/neurolabusc/Bru2Nii/releases/download/v1.0.20180303/Bru2_Linux.zip \
       && unzip Bru2_Linux.zip \
       && rm Bru2_Linux.zip" \
   --env PATH="\${PATH}:/opt/bru2" \
   --workdir="/opt" \
   --run="wget https://julialang-s3.julialang.org/bin/linux/x64/1.9/julia-1.9.3-linux-x86_64.tar.gz \
       && tar zxvf julia-1.9.3-linux-x86_64.tar.gz \
       && rm -rf julia-1.9.3-linux-x86_64.tar.gz" \
   --env PATH="\${PATH}:/opt/julia-1.9.3/bin" \
   --workdir="/opt" \
   --copy install_packages.jl "/opt" \
   --env JULIA_DEPOT_PATH="/opt/julia_depot" \
   --run="julia install_packages.jl" \
   --env JULIA_DEPOT_PATH="~/.julia:/opt/julia_depot" \
   --run="git clone --depth 1 --branch v0.5 https://github.com/astewartau/QSMxT-UI-2 QSMxT-UI" \
   --run="wget https://nodejs.org/dist/v14.17.0/node-v14.17.0-linux-x64.tar.xz \
       && tar xf node-v14.17.0-linux-x64.tar.xz \
       && rm node-v14.17.0-linux-x64.tar.xz" \
   --env PATH="\${PATH}:/opt/node-v14.17.0-linux-x64/bin" \
   --env DEPLOY_PATH="/opt/ants-2.3.4/:/opt/FastSurfer:/opt/QSMxT-UI" \
   --env DEPLOY_BINS="nipypecli:bet:dcm2niix:Bru2:Bru2Nii:tgv_qsm:julia:python3:python:pytest:predict_all.py:qsmxt:qsmxt-gui:dicom-sort:dicom-convert:nifti-convert"  \
   --env LC_ALL="C.UTF-8" \
   --env LANG="C.UTF-8" \
   --run="wget https://raw.githubusercontent.com/QSMxT/QSMxT/main/docs/container_readme.md -O /README.md" \
   --run="sed -i \"s/toolVersion/${toolVersion}/g\" /README.md" \
  > ${imageName}.${neurodocker_buildExt}

if [ "$1" != "" ]; then
   ./../main_build.sh
fi

# Explanation for Julia hack:
   # --env JULIA_DEPOT_PATH="/opt/julia_depot" \
   # --run="julia install_packages.jl" \
   # --env JULIA_DEPOT_PATH="~/.julia:/opt/julia_depot" \

   # The problem is that Julia packages install by default in the homedirectory
   # in singularity this homedirectory does not exist later on
   # so we have to set the Julia depot path to a path that's available in the image later
   # but: Julia assumes that this path is writable :( because it stores precompiled outputs there
   # solution is to to add a writable path before the unwritable path
   # behaviour: julia writes precompiled stuff to ~/.julia and searches for packages in both, but can't find them in ~/.julia and then searches in /opt/
   # if anyone has a better way of doing this, please let me know: @sbollmann_MRI (Twitter)

[Remi-Gau]

thanks let me check a few things

[Remi-Gau]

can reproduce with the most minimalist example:

neurodocker generate docker\        
   --base-image ubuntu:18.04 \
   --pkg-manager apt \
   --ants version=2.3.4 > tmp

So it seems like a new bug.

Noted that it does work with a more recent ANTs (2.4.3)

[astewartau]

Thank you! I'll see if I can update

[Remi-Gau]

Triggered a run of the tests workflows.

Clearly a lot of older ANTs versions are now failing:

You can see the combination of distro + ANTs version that are failing on the left column of this page. https://github.com/ReproNim/neurodocker/actions/runs/10249150842

In case this helps you choose.

[kaczmarj]

many of those older ants binaries are in a dropbox that i owned. i'm not sure why the 403 error is being raised... dropbox doesn't report any incidents on its status page.

however, i cannot log in to the dropbox right now.

[satra]

perhaps we can move them to zenodo or osf once access is restored, and update neurodocker. and more generally perhaps push static builds to zenodo or osf.

[Remi-Gau]

perhaps we can move them to zenodo or osf once access is restored, and update neurodocker. and more generally perhaps push static builds to zenodo or osf.

preference for zenodo

also some of the latest version of ANTs have their binaries on github so maybe we do not need to keep track of ALL their binaries