search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
24.04k stars 3.47k forks source link

what are our thoughts on explore logs patterns changing over time? #13108

Open trevorwhitney opened 5 months ago

trevorwhitney commented 5 months ago

this is a placeholder for a conversation, but as we aim to store timeseries data on patterns seen in the pattern ingester over longer periods of time, I think there are 2 problems we need to think about:

  1. Patterns will drift from one drain tree to another as the tree is reset, and we have too many series as it is. Do we have ideas on how to a) reduce the total number patterns, and b) make them applicable across drain trees/over time?
  2. The ability for a user to pin a pattern in time. If a user wants to "bookmark" a specific pattern, do we have thoughts on how to pin that drain tree in time?
matryer commented 5 months ago
  1. A pattern is, at the end of the day, a template string, right? You can always query for those, even if there are no logs. It may be that a pattern drifts out of use (say the devs change the log lines and it no longer matches that query) at which point it might be nice to show the logs up until when that pattern stopped.

Would we store metadata like 'first seen' and 'last seen'? Since patterns are ephemeral, if we've bookmarked a pattern that has gone out of fashion, we could show a notice saying 'Logs matching this pattern were last seen 2 months ago: View those logs or See current patterns'.

trevorwhitney commented 5 months ago

I think querying for an old pattern is not as much of a problem as showing patterns over time, for two reasons.

First, is it useful to show a bunch of little hills, where a pattern goes up for a little, then tails off, when looking over a long window (say ~1 wk)? Does that tell you anything about what's actually going on since it's missing the larger picture? This is where pattern evaluation at query time is helpful because all patterns are using the same drain tree over the whole window.

Second, that will explode cardinality to have a bunch of short lived patterns that, in reality, may be pretty similar. An idea @cyriltovena had for that is maybe storing derivatives of patterns by running drain on drain. The concern there is do you lose too much fidelity?