Add check for gsutil and put in sub-module

[Rachel-Alcraft]

Currently the call to gsutil allows failures which includes non-authetication as well as missing files. Change this to multiple calls to check authentication, and make it a sub-module.

This is necessary to understand if any nodes fail whn distributed on the clusters. There is a seperate consideration about how the files chunk if there are any missing.

[Rachel-Alcraft]

Do I need a servce account for the nodes: https://cloud.google.com/sdk/docs/authorizing: gcloud auth login --cred-file=CONFIGURATION_OR_KEY_FILE

[robertvi]

Ian's original suggestion was to use "gcloud auth application-default login" which seems to be for: "This command is useful when you are developing code that would normally use a service account but need to run the code in a local development environment where it's easier to provide user credentials" https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login Does that help?

[Rachel-Alcraft]

Have you managed to get it working on cluster nodes? It's fine to login on Myriad and get it working through nextflow locally, but I am struggling to provision the nodes - it doesn't re-use the local provision on the nodes as far as I can tell.

[robertvi]

It works on cs nodes using an interactive login, but that's cheating because I am still manually pasting in the auth string on the node that then runs the tasks. Testing proper qsub based non interactive job execution now...

[robertvi]

maybe put a https://www.nextflow.io/docs/latest/process.html#beforescript directive to a script that sets up auth for every process that needs it?

[Rachel-Alcraft]

Yes, I don't know what to put in the script! Have you got the nodes working?

[robertvi]

So far only got jobs submitted, nothing has reached front of the queue yet! Probably will all get cancelled when I move to the meeting and wifi cuts out briefly :P

[Rachel-Alcraft]

Deprecated effectively by the platform-specific config