prometheusMetrics millisSinceLastRepairForSchedule is stuck with default value and never updated

[aymeric-ledizes]

Project board link

I noticed issues with the millisSinceLastRepairForSchedule metric while checking the prometheusMetrics endpoint.

• From the UI I have a successful repair on reaper_db triggered by the schedule id “4e4092c0-1ce4-11ee-8972-6593653bfc28”
• However on the prometheusMetrics the value for millisSinceLastRepairForSchedule on the reaper_db schedule is still the default 1.688749312559E12 value. We can notice the schedule id is 4e4092c01ce411ee89726593653bfc28 without middle scores compared to the UI

io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="cassandratestaymeric",keyspace="reaperdb",scheduleid="4e4092c01ce411ee89726593653bfc28",} 1.688749312559E12

Environment

• cassandra: 3.11.13
• cassandra-reaper: V3.3.2
  • REAPER_STORAGE_TYPE="cassandra"
  • REAPER_DATACENTER_AVAILABILITY= "SIDECAR" (kubernetes)

┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: REAP-49

[mohammad-aburadeh]

Hi @adejanovski,

I'm facing the same issue. The metrics value is wrong for all keyspaces/schedules except the "reaperdb" keyspaces.

io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="prodcluster01",keyspace="etexndb",scheduleid="02d3ef60aeb911ed8fdc9d36331892b9",} 1.695235110636E12 io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="prodcluster01",keyspace="logdbexndb",scheduleid="de2c1a70be8a11ed8fdc9d36331892b9",} 1.695235110638E12 io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="prodcluster01",keyspace="reaperdb",scheduleid="91c70f10869e11ed89632b89086982d2",} 1.363413E7 io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="prodcluster01",keyspace="testexndb",scheduleid="c2ef6640368a11ee8fdc9d36331892b9",} 1.695235110643E12 io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="prodcluster01",keyspace="devexndb",scheduleid="8f6118e0a6da11ed8fdc9d36331892b9",} 1.695235110644E12 io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="prodcluster01",keyspace="lostandfoundexndb",scheduleid="71d729f002fa11ee8fdc9d36331892b9",} 1.695235110646E12 io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="prodcluster01",keyspace="prodexndb",scheduleid="2a914bc0218b11ee8fdc9d36331892b9",} 1.695235110648E12 io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="prodcluster01",keyspace="trainingexndb",scheduleid="9af12400c81011ed8fdc9d36331892b9",} 1.69523511065E12 io_cassandrareaper_service_RepairScheduleService_millisSinceLastRepairForSchedule{cluster="prodcluster01",keyspace="uatexndb",scheduleid="d7701dc002fb11ee8fdc9d36331892b9",} 1.695235110651E12

Can we have this issue fixed ASAP?

Regards Mohammad