search

fermitools / declad

BSD 3-Clause "New" or "Revised" License
1 stars 2 forks source link

Clean up config regarding datasets #25

Closed marcmengel closed 3 months ago

marcmengel commented 3 months ago

Due to rules in Rucio, Declad is configured to always add files into a dataset, and to create datasets. However, there are currently separate rucio and metacat dataset config entries, and this is confusing because the datasets should be defined the same in both systems. So I'm proposing we have a single config entry for this: datasettemplate: "name${field.first}_${field.second}..." and include a call out to an object that lets the experiment "custom" module: a) add to the available fields for formatting b) keep track of how many files it has added to a given dataset to limit dataset size.

This I think will implement everything that Dune needs, without an overly complicated config for smaller use.

marcmengel commented 3 months ago

Upon reflection, since there is already a rucio.dataset_did_template in the config, we should keep that for backwards compatability, and the metacat_dataset item is only defined in the per-experiment custom code, in the metacat_datset() callout, which we are no longer calling, in favor of using the rucio dataset name also for metacat, we have effectively already dropped support for the metacat_dataset() callout and config item.

marcmengel commented 3 months ago

So that leaves including the template_tags callout also on the rucio.dataset_did_template expansion, and we should be in business.

marcmengel commented 3 months ago

Calling this resolved with #27