We had an issue where a client was mis-configured, and was sending line format metrics to the pickle relay instead of the line relay. These are properly rejected: bucky-pickle-relay reads the first 4 bytes as the size, and as that will be ASCII data from the metric itself then the size will be a very large decimal value. The default logging in this case will only give the IP and ephemeral port, which are useless in many situations. I added this debugging code to help troubleshoot. It will log the size - which for line format data is just the first 4 bytes of the metric itself - along with the next 128 bytes.
Using some invented data, the logging looks like this:
2016/01/01 01:23:45 10.0.0.1:54321 attempting to send 1234567890 bytes and is too large, aborting
2016/01/01 01:23:45 Oversized pickle! size: host, data: name.application.metric.somevalue 0.123456
hostname.application.met
The raw metric data can make it much easier to determine what client is mis-configured.
jjneely
commented
8 years ago
We had an issue where a client was mis-configured, and was sending line format metrics to the pickle relay instead of the line relay. These are properly rejected: bucky-pickle-relay reads the first 4 bytes as the size, and as that will be ASCII data from the metric itself then the size will be a very large decimal value. The default logging in this case will only give the IP and ephemeral port, which are useless in many situations. I added this debugging code to help troubleshoot. It will log the size - which for line format data is just the first 4 bytes of the metric itself - along with the next 128 bytes.
Using some invented data, the logging looks like this:
The raw metric data can make it much easier to determine what client is mis-configured.