Changes required for Illumina DRAGEN support

[vigunase]

Note: this requires more requirements gathering and review.

1. Consider adding a generic backend_parameter called "docker_host_configuration", with value "dragen_v1.0.0"
2. Create a new folder in the default storage accounts "configuration" container, called "docker_host_configuration" with a file named "dragen_v1.0.0.txt" in it (blob name: docker_host_configuration/dragen_v1.0.0.txt)
3. The sample file shall contain settings including any/all of the following:

start_task_url docker_run_parameters batch_image_offer batch_image_publisher batch_image_sku batch_image_version batch_node_agent_sku_id

Notes on each of these:

• TES should download the start_task_url, and create a new file for it (as is implemented today for start tasks)
• Docker run parameters should be sanitized to prevent malicious usage, and then inserted into the "docker run" command that's used for the executor image

Also consider:

• add folder in TES project to contain vendor-specific configurations, where each vendor has their own subfolder, and each configuration/version has its own leaf folder containing the above configuration file plus optional start task script. The script file can be named by convention (e.g. "docker_host_startup.sh") so there is no need to specify the url/name in the config file.
• embed these files in TES container, no need to copy them to storage
• WDL only needs to specify the name of the configuration - name of the config file w/o extension
• if customers want to see the vendor provided configuration, they can see it in this repository

[BMurri]

This requires more work

[olesya13]

Depends on https://github.com/Azure/Batch/issues/131

[vsmalladi]

@BMurri https://github.com/Azure/Batch/issues/131 has been closed and resolved. Can we move this to 4.0 release?

[BMurri]

@vsmalladi I don't think it will make 4.0 (unless we continue to stretch it out) but it definitely should be in 4.1 IMO.

I need to migrate the PRs contents into the new multi-repo structure we now have, and rebase it on all the latest changes. That's going to take a bit of time.

[vsmalladi]

@BMurri Okay cool. Lets aim for 4.1 then.

[ngambani]

@vsmalladi @BMurri this looks like a long pending issue open since 2021, are we actively working on this or can this be closed?

[BMurri]

@ngambani I'm pretty sure DRAGEN in CoA is dead at this point but @vsmalladi would know that situation more. I linked this with the GPU and NVMe issues to remind me of the generalized structure I had built (and never merged) for this issue because a somewhat lighter weight version would facilitate both of those issues especially if we did them together. That's the same reason I haven't closed the branch associated with this issue (to more easily reuse parts rather than completely recreate the solution).

[vsmalladi]

I would like to add this funtionality. Let me circle with Illumina. @BMurri can we add this general functionality to coa/tes?

[BMurri]

@vsmalladi depends on them. We need some sort of supported docker image that includes the tools. Note that Azure has retired CentOS 7 AFAIK so this will require some work on their part, honestly. It's OK by me if they require public internet access (aka no support for completely private task compute node deployments) but they would need to specify the minimum port & domain-or-ip-addresses that would have to be permitted for the compute nodes to access (to ensure those who wish to deploy with this can set it up).

[vsmalladi]

They want to release this. Will have a technical dive with Illumina for what is needed.