Close Rucio output datasets when workflow finishes

DUNE / dist-comp

Action items for DUNE distributed computing, and common scripts that are used.

2 stars 0 forks source link

Close Rucio output datasets when workflow finishes #172

Open dougbenjamin opened 4 months ago

dougbenjamin commented 4 months ago

As part of justIN workflow system - Please close the output datasets at the end of a workflow stage. Here is the link to the Rucio API - https://rucio.cern.ch/documentation/client_api/didclient/#close

Our RSE's are filling up with empty replicas because the datasets are not closed at the time that the last file replica is removed from a RSE.

dougbenjamin commented 4 months ago

What is the status of this?

Andrew-McNab-UK commented 4 months ago

As I mentioned before, it will take a while since it requires some significant internal changes to keep track of whether it has succeeded and retry and fit in with all the changes requested by DM. So not something we can deployed during the current freezes for data taking.