Azure / batch-shipyard

Simplify HPC and Batch workloads on Azure
MIT License
277 stars 121 forks source link

Environment variables from task factory's placeholder {0}, {1}, ... #302

Open maroony opened 5 years ago

maroony commented 5 years ago

I would like to use the task factorys placeholder variables {0} as described here to set environment variables in my jobs.yaml file:

job_specifications:
- id: myID
  environment_variables:
    FOO: {0}

At the moment I do something like this:

tasks:
  - docker_image: myImage
    command: >-
      sh -c "
      export FOO={0};
      run.sh
      "

The problem ist, that I can not use ${FOO} in paths for output_data:

output_data:
      azure_storage:
      - storage_account_settings: my_storage_account
        remote_path: /my/path/${FOO}/
        local_path: ${AZ_BATCH_NODE_SHARED_DIR}/here/are/my/files/${FOO}

It seems, that the placeholder variable {0} is only evaluated within the task command. Is there another way to "promote" the values of {0} so that I can use them everywhere in the shipyard's yaml files?

maroony commented 5 years ago

In the meantime I extract the information from the environment variable SHIPYARD_USER_CMD. So I do something like this:

    output_data:
      azure_storage:
      - storage_account_settings: my_storage_account
        remote_path: /my/path/$(tmp=${SHIPYARD_USER_CMD##*=}; echo ${tmp%%;*})
        local_path: ${AZ_BATCH_NODE_SHARED_DIR}/here/are/my/files/$(tmp=${SHIPYARD_USER_CMD##*=}; echo ${tmp%%;*})