Prometheus - live stats

[lenisko]

Expose live status to prometheus using looped queries.

Exposed metrics:

• gyms - current gyms gauge grouped by team_name, in_battle
• raids - current raids gauge grouped by level
• incidents - current incidents gauge grouped by kind, confirmed
• lures - current lures gauge grouped by type
• quests - current quests gauge grouped by ar

Disable by default along prometheus integrity, config defaults as below:

live_stats = false
live_stats_sleep = 120

Solves https://github.com/UnownHash/Golbat/issues/178

[lenisko]

Cost of all queries. Take a note that I have rather big amount of data in there.

INFO 2023-12-04 15:01:48 [Prometheus] LiveStats fetched in: 285.502307ms
INFO 2023-12-04 15:01:58 [Prometheus] LiveStats fetched in: 292.687966ms
INFO 2023-12-04 15:02:08 [Prometheus] LiveStats fetched in: 287.242653ms
INFO 2023-12-04 15:02:19 [Prometheus] LiveStats fetched in: 281.713844ms
INFO 2023-12-04 15:02:29 [Prometheus] LiveStats fetched in: 328.766727ms
INFO 2023-12-04 15:02:39 [Prometheus] LiveStats fetched in: 313.814474ms
INFO 2023-12-04 15:02:50 [Prometheus] LiveStats fetched in: 301.160212ms
INFO 2023-12-04 15:03:00 [Prometheus] LiveStats fetched in: 324.496514ms
INFO 2023-12-04 15:03:10 [Prometheus] LiveStats fetched in: 287.65345ms
INFO 2023-12-04 15:03:21 [Prometheus] LiveStats fetched in: 287.67487ms
INFO 2023-12-04 15:03:31 [Prometheus] LiveStats fetched in: 300.200841ms
INFO 2023-12-04 15:03:41 [Prometheus] LiveStats fetched in: 273.209768ms
INFO 2023-12-04 15:03:51 [Prometheus] LiveStats fetched in: 284.656702ms
INFO 2023-12-04 15:04:02 [Prometheus] LiveStats fetched in: 278.180243ms
INFO 2023-12-04 15:04:12 [Prometheus] LiveStats fetched in: 306.21181ms
INFO 2023-12-04 15:04:23 [Prometheus] LiveStats fetched in: 504.111382ms
INFO 2023-12-04 15:04:33 [Prometheus] LiveStats fetched in: 303.368251ms
INFO 2023-12-04 15:04:43 [Prometheus] LiveStats fetched in: 297.79557ms
INFO 2023-12-04 15:04:53 [Prometheus] LiveStats fetched in: 300.593412ms
INFO 2023-12-04 15:05:04 [Prometheus] LiveStats fetched in: 284.54512ms

[jfberry]

I think 10s is very aggressive for a default refresh rate, considering that every other query that golbat makes has been tuned and re-tuned to be as lightweight as possible. My view would be that 2 minutes would likely be a more suitable default

[lenisko]

Well yes. We could speed them up by adding some more indexes, but not sure if it's worth. Bumped default to 2m.

[jfberry]

It would likely be better to complete the in-memory pokestop and gym access and count based on state transition if this proves to be a problem. Each of these queries would likely require a different and specialised index

[lenisko]

Here are the counts of rows I have in tables:

• gym: 170633
• incidents: 161243
• pokestop: 747035

queries took arpox:

• gyms: 0.1996s
• raids: 0.0908s
• incidents: 0.1502s
• lures: 0.0014s

But in the end it all depends on specs, configuration, amount of data and few other factors. Personally, I think those are quick enough for now with high default.

[jfberry]

These reads will be causing locks (perhaps escalated to table wide locks) which block all table updates etc for a vanity metric, so it is appropriate that they are not repeated too frequently. The gym scan, for example, may well block all GMO updates.

[lenisko]

@jfberry I might as well set a default to off, what do you think?

[jfberry]

I think that would make sense. If these vanity metrics are not important enough for someone to activate, then they probably should not be running

[lenisko]

Considering there's close to no-cost of running prometheus integration itself, this should stay off by default. Done!