Enhancement: Implement monitoring for suspicious replica recoverer

[haozturk]

Enhancement Description

We need monitoring for the actions that replica recoverer daemon takes

Use Case

• to see what's the impact of this daemon, i.e. how many replicas it "fixes"
• if a user reports corrupt replicas, we can check this monitoring to see if it's caught by this machinery or not

Possible Solution

I think we simply need the suspicious replicas that the daemon processes (file name and RSE or simply PFN) and the action it took (ignore, create rule, declare bad or declare temporary unavailable). Ideally these should be pushed to rucio event monitoring.

Related Issues

No response

[haozturk]

A high level monitoring is already available in FTS monitoring [1]. Rucio uses Recovery activity for the transfers that replace bad replicas with healthy ones. It's already visible that this activity picked up after we enabled the daemon.

[1] https://monit-grafana.cern.ch/d/mtQFDScGk/cms-fts-metrics?from=1730234072240&orgId=11&to=1731493015953&var-activity=Recovery&var-bin=1h&var-dst_rse=All&var-fts_server=All&var-group_by=dst_rse&var-src_rse=All&var-vo=cms&var-protocol=All&viewPanel=11&var-auth_method=All