search

UpstreamDataInc / goosebit

A simplistic, opinionated remote update server implementing hawkBit™'s DDI API.
https://goosebit.rtfd.io
Apache License 2.0
12 stars 2 forks source link

Telemetry prevents running multiple instances #72

Closed easybe closed 3 weeks ago

easybe commented 4 weeks ago

https://github.com/UpstreamDataInc/goosebit/blob/c787cb4a30ab1e64daeb50410fed11ffccdd9750/goosebit/telemetry/prometheus.py#L10

This does not work when running multiple instances e.g. with gunicorn workers. We need to keep concurrency in mind.

b-rowan commented 4 weeks ago

What's the best way to do this you think? It seems like its an issue with running the actual prometheus client, is there a way to hook it into fastAPI directly? How do those metrics work with multiple worker instances?

https://github.com/UpstreamDataInc/goosebit/blob/c787cb4a30ab1e64daeb50410fed11ffccdd9750/goosebit/telemetry/prometheus.py#L2

easybe commented 4 weeks ago

I have no idea. I would assume that one shared Prometheus collector instance should be enough. But, I don't know how it will collect the metrics.

For now, an option to disable the feature (or temporarily remove it) would do the trick.

b-rowan commented 4 weeks ago

Ill add that to my settings PR.