New introspection `workflow.seqeraPlatform` scope for data from Seqera Platform

[ewels]

It'd be nice to have a new workflow introspection scope called workflow.seqeraPlatform to access metadata from Seqera Platform, when available.

For example:

workflow.seqeraPlatform.user.username
workflow.seqeraPlatform.user.name
workflow.seqeraPlatform.user.email
workflow.seqeraPlatform.user.roles
workflow.seqeraPlatform.workspace.id
workflow.seqeraPlatform.workspace.name
workflow.seqeraPlatform.computeEnvironment.id
workflow.seqeraPlatform.computeEnvironment.name
workflow.seqeraPlatform.pipeline.name
workflow.seqeraPlatform.pipeline.repository
workflow.seqeraPlatform.pipeline.revision
workflow.seqeraPlatform.run.id

..etc. Can think of many more, or compare to the Seqera Platform API for standard fields.

If available, these could be used to set variables within the workflow config. There are many examples, a simple one would be the work directory:

workDir = {"/path/to/a/work/directory/${workflow.seqeraPlatform.user.username}/${workflow.seqeraPlatform.pipeline.name}/${workflow.runName}"}

Another one would be for notification emails (maybe more relevant in case of custom notification emails, like nf-core, but still useful as a simple example):

notification.to = workflow.seqeraPlatform.user.email

[pditommaso]

-1. it platform that needs to orchestrate nextflow, not the other way around

[bentsherman]

I believe the main use case was to apply resource labels based on platform metadata such as the run ID and user. These settings would be defined in the platform (e.g. CE-level config) but you still need to expose this metadata to the config via an implicit variable or environment variable

[pditommaso]

Then platform should provide the corresponding config

[ewels]

Then platform should provide the corresponding config

That's exactly what I'm suggesting. That platform provides the config to Nextflow, via workflow variables.

These should 100% be read-only, the same as the other workflow introspection variables.

[pditommaso]

It's not clear what we are trying to solve with this?

[ewels]

We want to allow Nextflow users to make use of metadata that comes from Seqera Platform, in a read-only manner, within Nextflow scripts.

Example use cases:

• Using a Platform username within a publishing file-path
• Sending custom emails from the pipeline to the email address of the Platform user

I can find more if needed by going back to the Platform customers who requested it if you'd like.

[ewels]

Here's the original feature request, though I think I've seen it elsewhere as well: https://feedback.seqera.io/feature-requests/p/nextflow-needs-to-be-able-to-access-metadata-from-the-tower

[FloWuenne]

We have recently run into a situation where we wanted to see if someone launched the pipeline using Seqera Platform enabled. We had to use session.config.navigate('tower.enabled') to make this work.

Having workflow introspection as described by @ewels above would have been very helpful for this use case!

[ewels]

Here's another feature request (with several others merged into it) which is also essentially asking for the same thing: https://feedback.seqera.io/feature-requests/p/support-for-a-distinct-set-of-variables-in-the-config-param-file-launch-dir-work

[ewels]

Discussion in the Nextflow weekly:

• Would probably be nicer to have this interpolation within the Platform - part of the launch form parsing, rather than within Nextflow. These could then be used in parameters to go into Nextflow, if needed. I think this is what the above feature request is asking for.
  • Pro: Interpolation can then happen in any field, including those outside of the config (eg. work directory)
  • Con: The config could then have platform variables, nextflow variables and bash variables. Gets confusing which is which and how to parse