nebfield
commented
8 months ago Running docker in docker is experimental in nextflow. Getting docker in docker working in general can be quite tricky. I've never tried to get it working myself, so I can't offer much advice, sorry. Can you run singularity on the HPC? That normally solves most of these types of problems.
If you have to use docker I'd recommend installing all of the dependencies inside one big container, including:
- nextflow
- a cloned copy of the repository
- plink2
- pgscatalog_utils
Version information for each dependency is available here.
and launch the workflow without using an execution profile (i.e. omit -profile).
Fiwx
Description of the bug
I am going to use pgsc_calc in a study which will use HPC resources. This requires Dockerizing the pipeline I'm using for the study. The pipeline includes a bunch of pre- and post-processing and imputation, and of course pgsc_calc for calculating the polygenic scores.
I am getting a "/bin/bash: .command.run: No such file or directory" error, even though .command.run exists.
Extra information:
First, I ssh into my Ubuntu image:
Next, I enter the image that I previously built:
Inside of Docker (which uses an Ubuntu base):
Installing Docker within my Docker:
Docker is installed: docker --version → Docker version 25.0.3, build 4debf41
However, running
nextflow run pgscatalog/pgsc_calc -profile test,dockerat this stage results in:We can fix this, by doing this: Starting my (outer) Docker:
Within outer Docker:
Now: docker version --format '{{.Server.APIVersion}}' → 1.44
Attempting to run
nextflow run pgscatalog/pgsc_calc -profile test,dockernow results in:Then the 2nd time it runs, we simply get:
But, there is a .command.run in the work directory. The file
/bin/bash: /exact/specific/path/to/work/.command.runlooks normal and has the script within it..command.log says:
Let’s start over and open up the outer container and do things right the first time.
Now, inside the Docker:
We still get this error:
Inside
/home/ubuntu/workdir/fc, ls -a shows the following:. .. .command.begin .command.err .command.log .command.out .command.run .command.sh .exitcode NO_FILE PGS001229_22.txtThere are two .command.run files.
/home/ubuntu/workdir/fc/.command.runand also/home/ubuntu/workdir/23/.command.run.The contents of
/home/ubuntu/workdir/fc/.command.runare included.To get more information, I ran
NXF_DEBUG=2 nextflow run pgscatalog/pgsc_calc -profile test,docker, which gave this output:I ran:
root@72f41fdf1937:/home/ubuntu/workdir/work/work/37/de# ./.command.runResulting in:ls -l .command.run→-rw-r--r-- 1 root root 10654 Feb 12 17:49 .command.runI ran chmod u+x .command.run and ./.command.run again, in the /home/ubuntu/workdir/work/work/37/de directory, resulting in the error: /bin/bash: /home/ubuntu/workdir/work/work/37/de/.command.run: No such file or directory
/home/ubuntu/workdir/work/work/37/de/.command.run certainly 100% exists in this exact location. (Also, if I try to run something on my system that doesn’t exist, it just says “bash” not “/bin/bash”).
We've now confirmed:
.command.sh and .command.run seem to be both in their original, correct directories.
Command used and terminal output
No response
Relevant files
!/bin/bash
NEXTFLOW TASK: PGSCATALOG_PGSCALC:PGSCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)
set -e set -u NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x NXF_ENTRY=${1:-nxf_main}
nxf_tree() { local pid=$1
}
nxf_stat() { cpu_stat=() nxf_tree $1
}
nxf_mem_watch() { set -o pipefail local pid=$1 local trace_file=.command.trace local count=0; declare -a cpu_stat=(0 0 0 0 0 0 0 0) declare -a cpu_peak=(0 0 0 0 0 0 0 0) local mem_tot=$(< /proc/meminfo grep MemTotal | awk '{print $2}') local timeout local DONE local STOP=''
}
nxf_write_trace() { echo "nextflow.trace/v2" > $trace_file echo "realtime=$wall_time" >> $trace_file echo "%cpu=$ucpu" >> $trace_file echo "cpu_model=$cpu_model" >> $trace_file echo "rchar=${io_stat1[0]}" >> $trace_file echo "wchar=${io_stat1[1]}" >> $trace_file echo "syscr=${io_stat1[2]}" >> $trace_file echo "syscw=${io_stat1[3]}" >> $trace_file echo "read_bytes=${io_stat1[4]}" >> $trace_file echo "write_bytes=${io_stat1[5]}" >> $trace_file }
nxf_trace_mac() { local start_millis=$(nxf_date)
}
nxf_fd() { local FD=11 while [ -e /proc/$$/fd/$FD ]; do FD=$((FD+1)); done echo $FD }
nxf_trace_linux() { local pid=$$ command -v ps &>/dev/null || { >&2 echo "Command 'ps' required by nextflow to collect task metrics cannot be found"; exit 1; } local num_cpus=$(< /proc/cpuinfo grep '^processor' -c) local cpu_model=$(< /proc/cpuinfo grep '^model name' | head -n 1 | awk 'BEGIN{FS="\t: "} { print $2 }') local tot_time0=$(grep '^cpu ' /proc/stat | awk '{sum=$2+$3+$4+$5+$6+$7+$8+$9; printf "%.0f",sum}') local cpu_time0=$(2> /dev/null < /proc/$pid/stat awk '{printf "%.0f", ($16+$17)10 }' || echo -n 'X') local io_stat0=($(2> /dev/null < /proc/$pid/io sed 's/^.:\s*//' | head -n 6 | tr '\n' ' ' || echo -n '0 0 0 0 0 0')) local start_millis=$(nxf_date) trap 'kill $mem_proc' ERR
}
nxf_trace() { local trace_file=.command.trace touch $trace_file if [[ $(uname) = Darwin ]]; then nxf_trace_mac else nxf_trace_linux fi } nxf_container_env() { cat << EOF export PYTHONNOUSERSITE="1" export R_PROFILE_USER="/.Rprofile" export R_ENVIRON_USER="/.Renviron" export JULIA_DEPOT_PATH="/usr/local/share/julia" export PATH="\$PATH:/root/.nextflow/assets/pgscatalog/pgsc_calc/bin" EOF }
nxf_sleep() { sleep $1 2>/dev/null || sleep 1; }
nxf_date() { local ts=$(date +%s%3N); if [[ ${#ts} == 10 ]]; then echo ${ts}000 elif [[ $ts == %3N ]]; then echo ${ts/\%3N/000} elif [[ $ts == 3N ]]; then echo ${ts/3N/000} elif [[ ${#ts} == 13 ]]; then echo $ts else echo "Unexpected timestamp value: $ts"; exit 1 fi }
nxf_env() { echo '============= task environment =============' env | sort | sed "s/(.)AWS(.)=(.{6}).*/\1AWS\2=\3xxxxxxxxxxxxx/" echo '============= task output ==================' }
nxf_kill() { declare -a children while read P PP;do children[$PP]+=" $P" done < <(ps -e -o pid= -o ppid=)
}
nxf_mktemp() { local base=${1:-/tmp} mkdir -p "$base" if [[ $(uname) = Darwin ]]; then mktemp -d $base/nxf.XXXXXXXXXX else TMPDIR="$base" mktemp -d -t nxf.XXXXXXXXXX fi }
nxf_fs_copy() { local source=$1 local target=$2 local basedir=$(dirname $1) mkdir -p $target/$basedir cp -fRL $source $target/$basedir }
nxf_fs_move() { local source=$1 local target=$2 local basedir=$(dirname $1) mkdir -p $target/$basedir mv -f $source $target/$basedir }
nxf_fs_rsync() { rsync -rRl $1 $2 }
nxf_fs_rclone() { rclone copyto $1 $2/$1 }
nxf_fs_fcp() { fcp $1 $2/$1 }
on_exit() { exit_status=${nxf_main_ret:=$?} printf -- $exit_status > /home/ubuntu/workdir/fc.exitcode set +u docker rm $NXF_BOXID &>/dev/null || true exit $exit_status }
on_term() { set +e docker stop $NXF_BOXID }
nxf_launch() { docker run -i --cpu-shares 2048 --memory 6144m -e "NXF_TASK_WORKDIR" -e "NXF_DEBUG=${NXF_DEBUG:=0}" -u $(id -u) -e "HOME=${HOME}" -v /etc/passwd:/etc/passwd:ro -v /etc/shadow:/etc/shadow:ro -v /etc/group:/etc/group:ro -v $HOME:$HOME -v /root/.nextflow/assets/pgscatalog/pgsc_calc:/root/.nextflow/assets/pgscatalog/pgsc_calc -v /home/ubuntu/workdir:/home/ubuntu/workdir -w "$PWD" -u $(id -u):$(id -g) --platform linux/amd64 --name $NXF_BOXID ghcr.io/pgscatalog/pgscatalog_utils:v0.4.3 /bin/bash -c "eval $(nxf_container_env); /bin/bash /home/ubuntu/workdir/fc.command.run nxf_trace" }
nxf_stage() { true
stage input files
}
nxf_unstage() { true [[ ${nxf_main_ret:=0} != 0 ]] && return }
nxf_main() { trap on_exit EXIT trap on_term TERM INT USR2 trap '' USR1
}
$NXF_ENTRY
System information
Container engine: /usr/bin/docker OS: Ubuntu 22.04.3 LTS