nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.71k stars 622 forks source link

Dynamic path for publish section #5090

Closed Puumanamana closed 2 months ago

Puumanamana commented 3 months ago

Hello,

I'm trying out the new publish feature introduced in version 24.04.0, but wasn't able to figure out the right implementation in my case. In my current pipeline, a single workflow execution can process multiple datasets at the same time, so I need to publish each process output to the correct output directory (matching its dataset). I'm currently doing that using a closure that takes the dataset field of the map associated to each channel element. Here's a snapshot of my publishDir directive:

publishDir = [
  path: { meta, output -> "results/$meta.dataset/${task.process.split(':')[-1]}" }
]

With the new syntax, files can be published with the >> operator within a process or workflow in the publish section. However, I couldn't identify in the documentation how to add a dynamic directive with a closure applied to the objects being published (if it's still possible)

bentsherman commented 2 months ago

It is not currently supported, but we are going to add it in the final version as many users have requested it.

bentsherman commented 2 months ago

Closing in favor of #5103