search

ITISFoundation / osparc-ops-environments

osparc operations
MIT License
3 stars 6 forks source link

Metrics: total cpu consumption is missing values from Dec 1 2022 #108

Closed elisabettai closed 1 year ago

elisabettai commented 1 year ago

Running this query in prometheus (osparc.io): node_cpu_seconds_total_v2:nonidle_increase_over_nodes_12weeks_v2/3600

Shows only values up to Dec.1: image

Also I don't see that rule in "Status > Rules" in Prometheus. Might this be the problem?

Do you have an idea on how we can report a reasonable number for this quarter? As for reference, for last quarter (Sep-Nov.2022) we reported 10’800 CPU-hours

elisabettai commented 1 year ago

It looks like that the rule code_cpu_seconds_total_v2:nonidle_increase_over_nodes_12weeks_v2 disappeared from prometheus rules: https://monitoring.osparc.io/prometheus/rules

Did we maybe deleted it by mistake? Luckily we still have the rule node_cpu_seconds_total:nonidle_increase_over_nodes_12weeks

@mrnicegyu11, do you know what is the difference between the v2 and v1 rule?

mrnicegyu11 commented 1 year ago

I vaguely remember the v2 rules but I sadly dont remember clearly anymore if me or sylvain introduced them and for what reason :( I need to check if and why the vanished, please open an issue in the osparc-ops issue tracker if you want us to follow this up

elisabettai commented 1 year ago

@mrnicegyu11 this is the issue I transferred from osparc-simcore repo.

Could you re-create the rule node_cpu_seconds_total_v2:nonidle_increase_over_nodes_12weeks_v2/3600 that disappeared? @sanderegg doesn't remember creating it. Also, do you remember what is the difference between this "_v2" rule and the one that still exists (node_cpu_seconds_total:nonidle_increase_over_nodes_12weeks)?

mrnicegyu11 commented 1 year ago

Adressed in https://github.com/ITISFoundation/osparc-ops-environments/pull/110

elisabettai commented 1 year ago

Hi @mrnicegyu11 or @Surfict, I'd very soon need the missing data to be restored as you explained in the PR.

In order to make past data available in these new rules, please follow https://jessicagreben.medium.com/prometheus-fill-in-data-for-new-recording-rules-30a14ccb8467`

Ideally, it would be nice to have the data by the end of this week (for the quarterly reports). I hope that works for you.

mrnicegyu11 commented 1 year ago

done