quiltdata / nf-quilt

GNU General Public License v3.0
7 stars 1 forks source link

manual configuration #185

Open drernie opened 6 months ago

drernie commented 6 months ago

Right now we:

This is already brittle, and breaks completely for dynamically-generated URIs.

We should have the option to manually specify all the relevant parameters, and use those when available. This is how it was done for the original Seqera-built nf-quilt plugin:

quilt {
  packageName = 'genomes/yeast'
  registry = 's3://seqera-quilt'
  message = 'My commit message'
  meta = [pipeline: 'nf-core/rnaseq']
  force = false
}
w-osman-orion commented 6 months ago

I am assuming this would be for the package the data is published to. If so, it would be great to have the additional following options:

Also, I am not sure if this addresses the issue, as I was parsing the input parameters to dynamically create the URI's for both the inputs and outputs to the nf workflow.

drernie commented 6 months ago

You are right, it makes much more sense to have those application-specific configuration parameters inside your NextFlow configuration, rather than inside the URI.

The way this solves your dynamic URI problem is that right now nf-quilt tries to use the package name (from the path) to find the original output URI from the parameters file. Which fails because your inputs and outputs are from the same package are not in the parameters file.

This way:

  1. nf-quilt can look at the config struct, rather than the params file, and
  2. you can explicitly tell nf-quilt which output URI to use, and
  3. you can simply embed the metadata in the config file (which is the actual bug you are facing ,right?)
w-osman-orion commented 6 months ago

Yes, that is the bug I am facing.