search

fermitools / declad

BSD 3-Clause "New" or "Revised" License
1 stars 2 forks source link

using rucio dataset for metacat #31

Closed rlcee closed 2 weeks ago

rlcee commented 3 weeks ago

mover.py;400

        dataset_did = self.rucio_dataset_did( self.FileDesc, metadata)
        dataset_scope, dataset_name = dataset_did.split(":")

        self.log(f"dataset: {dataset_scope}:{dataset_name} vs {self.last_created_dataset}")
        if f"{dataset_scope}:{dataset_name}" != self.last_created_dataset:
            self.last_created_dataset = f"{dataset_scope}:{dataset_name}" 
            if mclient is not None:
                # create in metacat first...
                try:
                    mclient.create_dataset(f"{dataset_scope}:{dataset_name}")

the metacat dataset name should be different from the rucio dataset name (they are separate in the config file and logically). One use case we are interested in is no Rucio, which will fail in this code. We will be using all the custom functions, so please make sure they are active here. Thanks

marcmengel commented 3 weeks ago

DUNE wants the Metacat and Rucio datasets to always match... so I'll have to make this call the metacat_dataset call again and make the DUNE experiment specific one return the Rucio one...

rlcee commented 2 weeks ago

We will also enforce them to be the same, but I think logically, you can just set this up in the custom code. an experiment can have the rucio_dataset return metacat_dataset result or not (as I think you are saying above). I would prefer the equality not be enforced in the core code for all possible uses. Overall, I prefer things like multiple templates for paths, dataset, scopes, tags, and metadata should all just be left to the custom script, and simplify the config and core code.