Add ability to user `--overwrite-when-only-on-disk` flag

dynamic-entropy commented 8 months ago

Task description FTS has completed their round of internal discussions about the requested feature of the --overwrite-when-only-on-disk flag as a proposed solution, jointly by CMS DM and FTS. Here is the proposal put forward by FTS: https://cmsdm.web.cern.ch/documents/FTS3_CMS_DM_Overwrite-when-only-on-disk_proposal.pdf

A previous document, presented by @drkovalskyi at the DMDev meeting, explaining the issue and rationale behind the proposed solution can be found here, https://docs.google.com/document/d/1eUq9PM0TXcHpdzPyi3V8yxk1NrCqYHvJLMH0azwEhIQ/edit#heading=h.sdyjvwib0rb

Related issues This is not a directly related issue but I will do it together with this one: https://github.com/dmwm/CMSRucio/issues/596 Tape file exists errors reports: https://github.com/dmwm/CMSRucio/issues/447 Jira ticket for the same error: https://its.cern.ch/jira/browse/CMSTRANSF-567 There is a long list of tickets :sweat_smile: but these capture some initial reporting on the error and the core idea.

Action items

[x] Get agreement from dmwm and give confirmation to FTS with requested changes (if any)
[x] Create an issue in core rucio
[x] Add FTS config for tape-enabled SE
[x] Create PR for rucio https://github.com/rucio/rucio/issues/6581
[x] Deploy on integration and test on FTS pilot instance
[ ] Get it in production

FYI @ericvaandering @dciangot

labkode commented 2 months ago

Draft PR here: https://github.com/rucio/rucio/pull/6691/files Waiting for my FTS colleagues to implement it server side to test.

dynamic-entropy commented 2 months ago

dmwm / CMSRucio