Provide dashboard to get current state of Rucio service/API usage

[vkuznet]

Per our discussion in https://github.com/dmwm/WMCore/issues/11356#issuecomment-1299535083 I would like to request to implement timber like dashboard for Rucio service. In particular, we need the following type of information:

• the throughput stats from Rucio REST APIs, e.g. how many request per second it can sustain
• any average stats, e.g. mean/median, Rucio APIs take to process any given block/file
• timber like dashboard, e.g. see CMSWEB timber, which can provide current state of Rucio server load and provide breakdown of current clients based on API, IPs, DNs, etc. such that we may correlate activity of WMAgents and Rucio instances.

In order to make such dashboard, Rucio should provide relevant metrics from their log/APIs to the MONIT. @mrceyhun provided relevant https://github.com/dmwm/WMCore/issues/11356#issuecomment-1298450176 about data flow.

[amaltaro]

Thank you for creating this issue, Valentin. This will indeed be useful for debugging and also for future design considerations within WMCore.

[haozturk]

This will be very helpful for the OPS as well. The recent incident led to significant delay in production workflows and we couldn't figure out the root cause due to lack of monitoring.

[haozturk]

Hi all, I just want to mention that more and more workflows are getting affected by "the delay in Rucio injection" issue and we started to get complaints from the requestors side due to such delays.