Support for non-local file storage with GA4GH TES driver

[kellrott]

New feature

Currently, the GA4GH TES driver only support shared-file systems. Other distributed compute APIs, like the AWS batch driver, are able to manage object storage based non-shared file systems. The request is to bring the GA4GH TES driver up to speed with other Nextflow drivers.

Usage scenario

Using the GA4GH TES execution driver in a system where shared file system is not avalible.

Suggest implementation

Start with file copy pattern seen in: https://github.com/nextflow-io/nextflow/blob/master/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchFileCopyStrategy.groovy or https://github.com/nextflow-io/nextflow/blob/master/plugins/nf-azure/src/main/nextflow/cloud/azure/file/AzBashLib.groovy or https://github.com/nextflow-io/nextflow/blob/master/plugins/nf-google/src/main/nextflow/cloud/google/lifesciences/GoogleLifeSciencesFileCopyStrategy.groovy

But because TES is able to manage different types of non-local storage systems, this should be configurable.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vsmalladi]

@pditommaso can we reopen this?

[pditommaso]

Sure

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bentsherman]

@vsmalladi is testing this PR nextflow-io/nextflow#4195 with TES Azure, so if it works then it should fix this issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vsmalladi]

Taking a look for azure.