viash-io / viash

script + metadata = standalone component
https://viash.io
GNU General Public License v3.0
39 stars 2 forks source link

[BUG] Docker setup not executed when specifying only a nextflow runner. #731

Open DriesSchaumont opened 4 months ago

DriesSchaumont commented 4 months ago

What happened?

When using viash ns build --setup cb, the docker build is not being executed for a component that has a docker engine specified and only the nextflow runner (executable runner is missing).

Steps to reproduce

First make sure that the docker image does not exist

docker rmi dummycomponent/myscript:latest

Then run the following script to reproduce:

mkdir -p /tmp/test_bug
cd /tmp/test_bug
cat > _viash.yaml << 'HERE'
viash_version: 0.9.0-RC6

source: src
target: target
HERE

mkdir -p src/dummycomponent

cat > src/dummycomponent/config.vsh.yaml << 'HERE'

name: myscript
namespace: "dummycomponent"
arguments:
  - name: "--output"
    type: file
    direction: output
    default: "${id}"
resources:
  - type: python_script
    path: script.py
    text: |
      print("OK!", flush=True)
engines:
  - type: docker
    image: python:latest
runners:
  - type: nextflow
HERE

viash ns build --setup cb
nextflow run . -main-script ./target/nextflow/dummycomponent/myscript/main.nf \
-profile docker \
-c ./target/nextflow/dummycomponent/myscript/nextflow.config \
--publish_dir .

Expected behavior

The docker image should have been build by viash and should be available to nextflow.

Relevant log output

 N E X T F L O W   ~  version 24.04.2

Launching `./target/nextflow/dummycomponent/myscript/main.nf` [silly_goldberg] DSL2 - revision: 5a0941ba36

executor >  local (1)
[51/cb3c78] process > myscript:processWf:myscript_process (run)        [  0%] 0 of 1
[-        ] process > myscript:publishStatesSimpleWf:publishStatesProc -
ERROR ~ Error executing process > 'myscript:processWf:myscript_process (run)'

Caused by:
  Process `myscript:processWf:myscript_process (run)` terminated with an error exit status (125)

Command executed:

  # meta exports
  export VIASH_META_RESOURCES_DIR=".viash_meta_resources"
  export VIASH_META_TEMP_DIR="/tmp"
  export VIASH_META_NAME="myscript"
  # export VIASH_META_EXECUTABLE="$VIASH_META_RESOURCES_DIR/$VIASH_META_NAME"
  export VIASH_META_CONFIG="$VIASH_META_RESOURCES_DIR/.config.vsh.yaml"
  export VIASH_META_CPUS=1

  if [ ! -z ${VIASH_META_MEMORY_B+x} ]; then
    export VIASH_META_MEMORY_KB=$(( ($VIASH_META_MEMORY_B+999) / 1000 ))
    export VIASH_META_MEMORY_MB=$(( ($VIASH_META_MEMORY_KB+999) / 1000 ))
    export VIASH_META_MEMORY_GB=$(( ($VIASH_META_MEMORY_MB+999) / 1000 ))
    export VIASH_META_MEMORY_TB=$(( ($VIASH_META_MEMORY_GB+999) / 1000 ))
    export VIASH_META_MEMORY_PB=$(( ($VIASH_META_MEMORY_TB+999) / 1000 ))
    export VIASH_META_MEMORY_KIB=$(( ($VIASH_META_MEMORY_B+1023) / 1024 ))
    export VIASH_META_MEMORY_MIB=$(( ($VIASH_META_MEMORY_KIB+1023) / 1024 ))
    export VIASH_META_MEMORY_GIB=$(( ($VIASH_META_MEMORY_MIB+1023) / 1024 ))
    export VIASH_META_MEMORY_TIB=$(( ($VIASH_META_MEMORY_GIB+1023) / 1024 ))
    export VIASH_META_MEMORY_PIB=$(( ($VIASH_META_MEMORY_TIB+1023) / 1024 ))
  fi

  # meta synonyms
  export VIASH_TEMP="$VIASH_META_TEMP_DIR"
  export TEMP_DIR="$VIASH_META_TEMP_DIR"

  # create output dirs if need be
  function mkdir_parent {
    for file in "$@"; do
      mkdir -p "$(dirname "$file")"
    done
  }
  mkdir_parent "${id}"

  # argument exports
  export VIASH_PAR_OUTPUT="${id}"

  # process script
  set -e
  tempscript=".viash_script.sh"
  cat > "$tempscript" << VIASHMAIN
  ## VIASH START
  # The following code has been auto-generated by Viash.
  par = {
    'output': $( if [ ! -z ${VIASH_PAR_OUTPUT+x} ]; then echo "r'${VIASH_PAR_OUTPUT//\'/\'\"\'\"r\'}'"; else echo None; fi )
  }
  meta = {
    'name': $( if [ ! -z ${VIASH_META_NAME+x} ]; then echo "r'${VIASH_META_NAME//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'functionality_name': $( if [ ! -z ${VIASH_META_FUNCTIONALITY_NAME+x} ]; then echo "r'${VIASH_META_FUNCTIONALITY_NAME//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'resources_dir': $( if [ ! -z ${VIASH_META_RESOURCES_DIR+x} ]; then echo "r'${VIASH_META_RESOURCES_DIR//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'executable': $( if [ ! -z ${VIASH_META_EXECUTABLE+x} ]; then echo "r'${VIASH_META_EXECUTABLE//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'config': $( if [ ! -z ${VIASH_META_CONFIG+x} ]; then echo "r'${VIASH_META_CONFIG//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'temp_dir': $( if [ ! -z ${VIASH_META_TEMP_DIR+x} ]; then echo "r'${VIASH_META_TEMP_DIR//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'cpus': $( if [ ! -z ${VIASH_META_CPUS+x} ]; then echo "int(r'${VIASH_META_CPUS//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_b': $( if [ ! -z ${VIASH_META_MEMORY_B+x} ]; then echo "int(r'${VIASH_META_MEMORY_B//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_kb': $( if [ ! -z ${VIASH_META_MEMORY_KB+x} ]; then echo "int(r'${VIASH_META_MEMORY_KB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_mb': $( if [ ! -z ${VIASH_META_MEMORY_MB+x} ]; then echo "int(r'${VIASH_META_MEMORY_MB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_gb': $( if [ ! -z ${VIASH_META_MEMORY_GB+x} ]; then echo "int(r'${VIASH_META_MEMORY_GB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_tb': $( if [ ! -z ${VIASH_META_MEMORY_TB+x} ]; then echo "int(r'${VIASH_META_MEMORY_TB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_pb': $( if [ ! -z ${VIASH_META_MEMORY_PB+x} ]; then echo "int(r'${VIASH_META_MEMORY_PB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_kib': $( if [ ! -z ${VIASH_META_MEMORY_KIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_KIB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_mib': $( if [ ! -z ${VIASH_META_MEMORY_MIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_MIB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_gib': $( if [ ! -z ${VIASH_META_MEMORY_GIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_GIB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_tib': $( if [ ! -z ${VIASH_META_MEMORY_TIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_TIB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_pib': $( if [ ! -z ${VIASH_META_MEMORY_PIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_PIB//\'/\'\"\'\"r\'}')"; else echo None; fi )
  }
  dep = {

  }

  ## VIASH END
  print("OK!", flush=True)
  VIASHMAIN
  python -B "$tempscript"

Command exit status:
  125

Command output:
  (empty)

Command error:
  Unable to find image 'dummycomponent/myscript:latest' locally
  docker: Error response from daemon: pull access denied for dummycomponent/myscript, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
  See 'docker run --help'.

Work dir:
  /tmp/test_bug/work/51/cb3c787af167a9c291cf961fa3710d

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Version

Possible solution

No response

Confirmation

Additional context

No response

Grifs commented 4 months ago

The issue seems to be with line 42 in ViashBuild.scala:

val exec_path = resources.mainScript.map(scr => Paths.get(output, scr.resourcePath).toString)

For Nextflow runners, there is no mainScript, so subsequent code is not executed ... but in itself that makes sense, because the code to generate the docker file would be in the bash wrapper, which we just established isn't there.