When Graylog receives messages, it records the message content and metadata in the journal for later stages to pick them up.
Due to external factors, such as load spikes, network constraints, or other downstream indexing issues, there can be a significant delay between when a message has been delivered to the server vs when it is sent off to indexing it.
This per-server processing latency is important to users and operators to understand the state and health of a system.
The alerting system already considers processing latency, but it is probably not properly exposed to users yet.
Why?
If the processing latency is very high, recent data received by Graylog servers won't be available for searching; thus, while the system is catching up, searches over recent time frames will not be accurate. Users need to be aware of this.
For operators, high processing latency can help when measuring a cluster or individual servers' health (current or future).
mpfz0r
commented
7 months ago
What?
When Graylog receives messages, it records the message content and metadata in the journal for later stages to pick them up. Due to external factors, such as load spikes, network constraints, or other downstream indexing issues, there can be a significant delay between when a message has been delivered to the server vs when it is sent off to indexing it.
This per-server processing latency is important to users and operators to understand the state and health of a system.
The alerting system already considers processing latency, but it is probably not properly exposed to users yet.
Why?
If the processing latency is very high, recent data received by Graylog servers won't be available for searching; thus, while the system is catching up, searches over recent time frames will not be accurate. Users need to be aware of this.
For operators, high processing latency can help when measuring a cluster or individual servers' health (current or future).
Your Environment