enkore
commented
8 years ago Instead of federation we (@rehwanne, @enkore) are considering a different approach. Every vassal gets a singular thread/process somewhere (e.g. via uwsgidecorators or as a mule) that pings a central django microservice (with no datastore or anything like that) every second or so. The django service in turn updates a config file read by prometheus.
This fulfills the original design requirements:
- no need to update promtheus configuration when workers are added/removed
- no need for explicit prometheus port configuration
- self-updating/recovering
https://prometheus.io/docs/operating/configuration/#%3Cfile_sd_config%3E
Currently queried through /metrics over network. The problem is that uWSGI has multiple workers for each server and via the dispatch mechanism may serve up different works (and thus stats).
Workaround-idea-ish: Have a mule for each server that internally queries and aggregates /metrics of each worker. Should be possible by hacking on prometheus-server.
Another approach would be to export via random ports and do some kind of service discovery.