nextflow-io / nf-nomad

Hashicorp Nomad executor plugin for Nextflow
https://nextflow-io.github.io/nf-nomad/
Apache License 2.0
2 stars 3 forks source link

`bin` dir scripts don't seem to work #59

Closed matthdsm closed 2 months ago

matthdsm commented 3 months ago

VERSION

0.1.1

CMD

nextflow run nf-core/fetchngs -profile test -c nomad.config  -w /scratch --outdir ./fetchngs

Error

ERROR ~ Error executing process > 'NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO (SRR14709033)'

Caused by:
  Process `NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO (SRR14709033)` terminated with an error exit status (127)

Command executed:

  echo SRR14709033 > id.txt
  sra_ids_to_runinfo.py \
      id.txt \
      SRR14709033.runinfo.tsv \

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO":
      python: $(python --version | sed 's/Python //g')
  END_VERSIONS

Command exit status:
  127

Command output:
  (empty)

Command error:
  .command.sh: line 4: sra_ids_to_runinfo.py: command not found

Work dir:
  /scratch/f2/9a53d5bf94fffd7ad3e4fa6a61d9af

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

CONFIG

plugins {
     id 'nf-nomad@0.1.1'
}

process {
     executor = "nomad"
     docker.enabled = true
}

nomad {
     client {
          address = "http://nomad.server"
          token   = "token"
     }

     jobs {
          deleteOnCompletion = false
          namespace = "nextflow"
          datacenters = ['dc']

          volumes = [
               { type "csi" name "nf_scratch_volume" path "/scratch"}
          ]
     }
}
matthdsm commented 3 months ago

reproducible on the azure server

matthdsm commented 3 months ago

Data in assets also isn't available, effectively breaking multiqc in all nf-core pipelines

matthdsm commented 3 months ago

I guess we're still missing something like this: https://github.com/nextflow-io/nextflow/blob/059151a6bad40baedcaaf2b31fa455a951bd420b/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchExecutor.groovy#L89-L98

matthdsm commented 3 months ago

Setting NXF_ASSETS to a directory mounted by nomad fixes this issue

abhi18av commented 3 months ago

Yeah, this makes it a good candidate (with workDir ) to perhaps dynamically set this at runtime 🤔

That being said, shall we close this issue?

abhi18av commented 2 months ago

Closing this as the best way to test the plugin is through https://github.com/nextflow-io/nf-nomad/pull/62#issuecomment-2212482769

And for a released build, the same process should be followed after adding the relevant NXF_* parameters.