Multi-instance task coordination command working directory issue

[rupertnash]

If you add a resource file to the coordination command for a multi-instance task then this is not downloaded to the task working directory. Instead, it is added to the parent of this (where the task stderr.txt etc are placed). This wouldn’t be a problem except that the working directory for coordination command execution is the task working directory. (Checked by setting my command string to be pwd && sh ../coordination.sh).

As a work-around you can just cd .. first, but it's quite annoying.

This is for the agent SKU ID "batch.node.centos 7", in case this is specific to that.

[matthchr]

I will look into this. I am not sure if this is intended or a bug.

[matthchr]

I investigated this and determined that this behavior is expected at least currently.

Here's the reasoning behind it:

1. It is intended that we download the common resource files to the task directory (and not the task working directory). This was primarily to dodge the possibility of conflicts between master task resource files and common resource files, and to put them into different folders for clarity. Also to more reasonably support multiple subtasks on the same node (which could then share files as opposed to downloading the same files into two different locations on the node).
2. It is the case that the coordination command line runs from the task working directory though, which means that you do have to do cd .. OR run the executable using the AZ_BATCH_TASK_SHARED_DIR environment variable.

Does using $AZ_BATCH_TASK_SHARED_DIR make sense in your use case?

[rupertnash]

Thanks for this Matt. Could this please be documented somewhere?

[matthchr]

Yes, we should definitely document this more clearly than we do. I've made a note and we will update our documentation shortly.