the GCP load balancer health checker makes about 4 requests per second to /heartbeat. This ends up being a massive portion of the total events that we send to honeycomb (especially for staging), and are using up most of our quota. They're mostly "boring" though. The requests are all exactly the same and don't contribute much to analysis that we might want to do.
For situations like this, Honeycomb lets you sample events at the origin. This adds custom sampler logic that lets us just drop 99% of the "boring" heartbeat requests, while keeping everything else (including any heartbeat requests that have a non-200 response).
If we end up adding more complicated sampling logic, we'll probably want to pull the sample rate, etc out into ansible variables. For now, I think it's fine to just hard-code them in there.
the GCP load balancer health checker makes about 4 requests per second to
/heartbeat. This ends up being a massive portion of the total events that we send to honeycomb (especially for staging), and are using up most of our quota. They're mostly "boring" though. The requests are all exactly the same and don't contribute much to analysis that we might want to do.For situations like this, Honeycomb lets you sample events at the origin. This adds custom sampler logic that lets us just drop 99% of the "boring" heartbeat requests, while keeping everything else (including any heartbeat requests that have a non-200 response).
This is based on sample code from here: https://docs.honeycomb.io/getting-data-in/python/beeline/#sampling-events
If we end up adding more complicated sampling logic, we'll probably want to pull the sample rate, etc out into ansible variables. For now, I think it's fine to just hard-code them in there.