Add metacat dataset creation to DeclaD

[StevenCTimm]

in mover.py the declaD currently has code to declare a Rucio dataset at line 475-476 based on the format in the rucio_dataset_did method, drawing from the dataset_did_template in the config file.

What we need the declad to now do is to also declare a metacat dataset with the same namespace and name. this could be inserted into the metacat declaration code just above it.. lines 449-450 or thereabouts.

This is essential to get the declad up and working with rucio v33.

[StevenCTimm]

Marc supplied a patch, I tried it, it works.

[StevenCTimm]

actually there is another issue.. if there is more than one file that is supposed to go into the same metacat data set it is trying to create the metacat dataset every time, and failing to do so and throwing an exception. Believe this can be solved with a simple try/except

[StevenCTimm]

I see there already is such a structure around this code.

Traceback (most recent call last): File "/home/np04data/miniconda3/lib/python3.10/site-packages/pythreader/task_queue.py", line 210, in run result = task.run() File "/home/np04data/declad/declad_2.0.1/mover.py", line 480, in run mclient.create_dataset(f"{dataset_scope}:{dataset_name}") File "/home/np04data/miniconda3/lib/python3.10/site-packages/metacat/webapi/webapi.py", line 471, in create_dataset return self.post_json(url, params) File "/home/np04data/miniconda3/lib/python3.10/site-packages/metacat/webapi/webapi.py", line 215, in post_json response = self.send_request("post", uri_suffix, data=data, headers=headers, stream=True) File "/home/np04data/miniconda3/lib/python3.10/site-packages/metacat/webapi/webapi.py", line 164, in send_request raise AlreadyExistsError(url, response) metacat.webapi.webapi.AlreadyExistsError: Already exists

Above is the exception we are getting.

I made the following changes:

from metacat.webapi import MetaCatClient, MCServerError from metacat.webapi.webapi import AlreadyExistsError

and now mover.py looks like this

            if mclient is not None:
                # create in metacat first...
                try:
                    mclient.create_dataset(f"{dataset_scope}:{dataset_name}")
                except metacat_client.MCServerError:
                    pass
                except metacat_client.AlreadyExistsError:
                    pass

---

This seems to handle multiple files correctly now

---

and added the following to mover.py

            if mclient is not None:
                # create in metacat first...
                try:
                    mclient.create_dataset(f"{dataset_scope}:{dataset_name}")
                except metacat_client.MCServerError:
                    pass
                except metacat_client.AlreadyExistsError:
                    pass