Bug in path resolution in local config file

[dcl10]

Description

When running the following command: WfExS-backend/WfExS-backend.py -L local-config.yml execute -W test-stage.yml I got the following error message:

schema_salad.exceptions.ValidationException: Not found: '/root//root/wfexs-backend-test_WorkDir/efb98299-cb1f-48f8-862e-7a8746bba1a4/workflow/workflows/sec-hutchx86.cwl'

The path resolution appears to have added an additional /root/ to the front of the path in local-config.yml (see below). When I changed the workDir to ./wfexs-backend-test_WorkDir, the execution appeared to proceed as expected and I saw this in the logging output:

materialized workflow repository (checkout 6d500ca1396283faae2ce5eebf778500dd8be2da): /root/wfexs-backend-test_WorkDir/f51c9984-8e43-49fa-a03b-8e683e884980/workflow

The path resolves as would be expected if I ran WfExS from /root.

Local config file

cacheDir: $HOME/wfexs-backend-test
crypt4gh:
  key: local_config.yaml.key
  passphrase: strive backyard dividing gumball
  pub: local_config.yaml.pub
tools:
  containerType: podman
  dockerCommand: docker
  podmanCommand: podman
  encrypted_fs:
    command: encfs
    type: encfs
  engineMode: local
  gitCommand: git
  javaCommand: java
  singularityCommand: singularity
  staticBashCommand: bash-linux-x86_64
workDir: $HOME/wfexs-backend-test_WorkDir

Stage file

workflow_id: https://raw.githubusercontent.com/HDRUK/hutch/main/workflows/sec-hutchx86.cwl
workflow_config:
  container: 'podman'
  secure: false
nickname: 'vas-workflow'
cacheDir: /tmp/wfexszn6siq2jtmpcache
crypt4gh:
  key: cosifer_test1_cwl.wfex.stage.key
  passphrase: mpel nite ified g
  pub: cosifer_test1_cwl.wfex.stage.pub
outputs:
  output_file:
    c-l-a-s-s: File
    glob: "output.json"
params:
  body:
    c-l-a-s-s: File
    url:
      - https://raw.githubusercontent.com/HDRUK/hutch/main/workflows/inputs/rquest-query.json
  is_availability: true
  db_host: "localhost"
  db_name: "hutch"
  db_user: "postgres"
  db_password: "example"

[jmfernandez]

I have realized several unrelated issues in the configuration files you have shared here meanwhile I was trying to reproduce the issue with current version.

The first one is that the local configuration file is using what it seems environment variables both in workDir and cacheDir. Environment variable substitution for file paths in the local configuration file is not going to be implemented, so I guess you put them there in order to avoid disclosing local paths, right?

The second one is that the sample stage file you have included is using container key under workflow_config, instead of containerType one. The validation does not complain because those blocks are not being constrained by an "additionalProperties": false.

For this last issue, you can see the automatically generated documentation here: https://github.com/inab/WfExS-backend/blob/main/development-docs/schemas/stage-definition_schema.md#workflow_config

Last, once I fixed these two issues (replacing the environment variable placeholders by their values by hand and using containerType key), staging seemed to properly work.

So, I don't know whether I was unable to reproduce the issue, or it was fixed due past commits.

[jmfernandez]

@dcl10 did you have a look at my previous answer of this issue? Is this issue still relevant?

[dcl10]

Hi @jmfernandez, I'm afraid I hadn't taken a look.

However, I've had some luck changing how I specified workflow_id to be like pypi.

On the first issue, thanks for the heads up about workDir and cacheDir. We used them simply because they were in one of the example workflows. What should we do instead for these fields? Simply not specify them?

For the second one, I found in the docs that it should have been containerType and have using that since.

I think this issue can be closed now, so I'll do that. Thank you for having a look for me - much appreciated 👍