We've had an issue where after a backward incompatible change in GitLab 16.0 in how it populates variables, DD_AGENT_HOST variable suddenly became an empty string, and all places where hot-shots is used to send metrics from CI stopped sending metrics, although there were no errors in logs, and it took a while to understand why things have broken. Because by reading logs, everything looked "normal".
I managed to figure it out by reading this comment:
We've had an issue where after a backward incompatible change in GitLab 16.0 in how it populates variables,
DD_AGENT_HOSTvariable suddenly became an empty string, and all places wherehot-shotsis used to send metrics from CI stopped sending metrics, although there were no errors in logs, and it took a while to understand why things have broken. Because by reading logs, everything looked "normal".I managed to figure it out by reading this comment:
https://github.com/brightcove/hot-shots/issues/197#issuecomment-792042173
and logging the value of
DD_AGENT_HOSTin CI and seeing it's empty.Relevant piece of code:
https://github.com/brightcove/hot-shots/blob/b4422446831b47873637b43ff6d111b1cef1b5d5/lib/statsd.js#L61
So it would be good to improve at least this particular aspect, but also in general the debugging is not always obvious, see https://github.com/brightcove/hot-shots/issues/257