search

RocketChat / Rocket.Chat

The communications platform that puts data protection first.
https://rocket.chat/
Other
40.16k stars 10.38k forks source link

Prometheus metrics are refreshed only once in an hour #12489

Open konfusator opened 5 years ago

konfusator commented 5 years ago

Description:

Steps to reproduce:

  1. Activate prometheus
  2. Go to http://localhost:9485/metrics (or whatever port is configured)
  3. Look at values like rocketchat_direct_messages_total or rocketchat_private_group_messages_total.

Expected behavior:

Get recent values for the metrics as found in https://rocket.example.com/admin/info after pressing the »Refresh«-button.

As getting the metrics may be an expensive operation and the metrics-URL is not protected a short caching time (about 5 minutes) might be reasonable.

Actual behavior:

Counters are constant for an hour and than updated. If the »Refresh«-button in the admin interface is pressed the prometheus counters are refreshed too.

Server Setup Information:

geekgonecrazy commented 5 years ago

I believe Prometheus reads from stats report that is generated once per hour instead of actual building real time. Definitely valid issue

Armadill0 commented 5 years ago

I just stumbled across this. Is there a possibility to configure the statistics refresh interval?

At least the statistics API endpoint (https://rocket.chat/docs/developer-guides/rest-api/miscellaneous/statistics/) has the possibility to refresh the data on the fly. Maybe the Prometheus logic can be rewritten to use this instead?

geekgonecrazy commented 5 years ago

most of the data provided by prometheus endpoint is the data of internal workings and method timing. Its not necessarily meant for all of this data, its just been exposed here.

As far as I know though there isn't a way to set interval. I've seen someone else have a cronjob that hit one of the stats endpoints to generate more frequently. Any stats generated will be the same stats the prometheus endpoint delivers.

jenskuespert commented 4 years ago

I'd like to ask for a configurable interval, too. Up to now this seems to be hard-coded in server/startup/cron.js:

                   SyncedCron.add({
                            name,
                            schedule(parser) {
                                    return parser.cron('12 * * * *');
                            },
                            job: generateStatistics,
                    });

It'd be really nice, if we could change this, because this statistics are used to report stuff to any module downstream...