nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.61k stars 605 forks source link

New introspection `workflow.seqeraPlatform` scope for data from Seqera Platform #5077

Open ewels opened 1 week ago

ewels commented 1 week ago

It'd be nice to have a new workflow introspection scope called workflow.seqeraPlatform to access metadata from Seqera Platform, when available.

For example:

workflow.seqeraPlatform.user.username
workflow.seqeraPlatform.user.name
workflow.seqeraPlatform.user.email
workflow.seqeraPlatform.user.roles
workflow.seqeraPlatform.workspace.id
workflow.seqeraPlatform.workspace.name
workflow.seqeraPlatform.computeEnvironment.id
workflow.seqeraPlatform.computeEnvironment.name
workflow.seqeraPlatform.pipeline.name
workflow.seqeraPlatform.pipeline.repository
workflow.seqeraPlatform.pipeline.revision
workflow.seqeraPlatform.run.id

..etc. Can think of many more, or compare to the Seqera Platform API for standard fields.

If available, these could be used to set variables within the workflow config. There are many examples, a simple one would be the work directory:

workDir = {"/path/to/a/work/directory/${workflow.seqeraPlatform.user.username}/${workflow.seqeraPlatform.pipeline.name}/${workflow.runName}"}

Another one would be for notification emails (maybe more relevant in case of custom notification emails, like nf-core, but still useful as a simple example):

notification.to = workflow.seqeraPlatform.user.email
pditommaso commented 1 week ago

-1. it platform that needs to orchestrate nextflow, not the other way around

bentsherman commented 1 week ago

I believe the main use case was to apply resource labels based on platform metadata such as the run ID and user. These settings would be defined in the platform (e.g. CE-level config) but you still need to expose this metadata to the config via an implicit variable or environment variable

pditommaso commented 1 week ago

Then platform should provide the corresponding config

ewels commented 2 days ago

Then platform should provide the corresponding config

That's exactly what I'm suggesting. That platform provides the config to Nextflow, via workflow variables.

These should 100% be read-only, the same as the other workflow introspection variables.