Open brettimus opened 11 months ago
So this might end up being pretty edge-case-y.
The original scenario that reproduced this for me was:
AUTOMETRICS_TRACKER=prometheus
enviornment variableinit
with the build informationMy working theory is that this issue will only occur if you call init
with a version, without also changing the tracker.
In such a case, the tracker would be recording two separate build_info metrics within a short period of time (less than the interval we use in our queries)
It begs the question: Could we use a concept of a "clearmode" internally on the build_info gauge?
Basically, last-in-wins for build information. If you record a build_info metric, it will remove any other previously recorded build_info metrics that your app is exporting.
This could pose other issues (I haven't fully thought this through yet), but it would make a stronger guard at the library level that against corrupting our build info queries.
When you set, e.g., a
version
viainit
, then I'm pretty sure we end up adding two gauges, one with aversion
label and one without a version label. These will both be set within a short period of time.The result is that our
group left
query that joins build info to function metrics will cause a 422 from prometheus, with an error like:The culprit is likely:
We initialize a tracker out of the box. When a user calls
init
, they effectively "re-initialize" the tracker with new build information, which sets a new build info gauge.Need to confirm though.