High cardinality labels

[sandstrom]

For many logging scenarios it's useful to lookup/query the log data on high cardinality labels such as request-ids, user-ids, ip-addresses, request-url and so on.

While I realize that high cardinality will affect indices, I think it's worth discussing whether this is something that Loki can support in the future.

There are a lot of logging use-cases where people can easily ditch full-text search that ELK and others provide. But not having easy lookup on log metadata, such as user-id or request-id, is a problem.

I should note that this doesn't necessarily need to be "make labels high cardinality", it could also be the introduction of some type of log-line metadata or properties, that are allowed to be high cardinality and can be queried for.

Example

Our services (nginx, application servers, etc) emit JSON lines[1] like this:

{ "timestamp" : "2018-12-24T18:00:00", "user-id" : "abc", "request-id" : "123", "message" : "Parsing new user data" }
{ "timestamp" : "2018-12-24T18:01:00", "user-id: " "def", "request-id" : "456", "message" : "Updating user record" }
{ "timestamp" : "2018-12-24T18:02:00", "user-id: " "abc", "request-id" : "789", "message" : "Keep alive ping from client" }

We ingest these into our log storage (which we would like to replace with Loki), and here are some common types of tasks that we currently do:

1. Bring up logs for a particular user. Usually to troubleshoot some bug they are experiencing. Mostly we know the rough timeframe (for example that it occurred during the past 2 weeks). Such a filter will usually bring up 5-200 entries. If there are more than a few entries we'll usually filter a bit more, on a stricter time intervall or based on other properties (type of request, etc).

2. Find the logs for a particular request, based on its request id. Again, we'd usually know the rough timeframe, say +/- a few days.

3. Looking at all requests that hit a particular endpoint, basically filtering on 2-3 log entry properties.

All of these, which I guess are fairly common for a logging system, require high cardinality labels (or some type of metadata/properties that are high cardinality and can be queried).

[1] http://jsonlines.org/

Updates

A few updates, since this issue was originally opened.

• This design proposal doesn't directly address high-cardinality labels, but solves some of the underlying problems.

  With LogQL you can grep over large amounts of data fairly easily, and find those needles. It assumes that the queries can run fast enough on large log corpora.

• There is some discussion in this thread, on making it easier to find unique IDs, such as trace-id and request-id

[sandstrom]

A thought:

An alternative to high cardinality labels could be to introduce a complement to labels called metadata, that is allowed to be high cardinality. Having two different things could allow us to impose other restrictions on the metadata type of values.

If grep is quick enough, maybe it would work to not index the metadata key/value pairs. But still allow them to be filtered (at "grep speed"). That would allow you to give these high-cardinality key-value pairs blessed UI, for easy filtering, whilst still avoiding the cost of indexing high cardinality fields.

[tomwilkie]

While I realize that high cardinality will affect indices, I think it's worth discussing whether this is something that Loki can support in the future.

The metadata we index for each streams has the same restrictions as it does for Prometheus labels: I don't want to say never, but I'm not sure if this is something we're ever likely to support.

An alternative to high cardinality labels could be to introduce a complement to labels called metadata, that is allowed to be high cardinality. But not having easy lookup on log metadata such as user-id or request-id may pose a problem.

Does the support for regexp filtering allow you to filter by a given user-id or request-id and achieve the same end as a second sets of non-index labels?

I'd agree its cumbersome, so perhaps adding some jq style language here to make this more natural for json logs would be better?

[sandstrom]

@tomwilkie It's all about speed, really. For our current ELK-stack — which we would like to replace with Loki — here are some common tasks:

1. Bring up logs for a particular user. Usually to troubleshoot some bug they are experiencing. Mostly we know the rough timeframe (for example that it occurred during the past 2 weeks). Such a filter will usually bring up 5-200 entries. If there are more than a few entries we'll usually filter a bit more, on a stricter time intervall or based on other properties (type of request, etc).

2. Find the logs for a particular request, based on its request id. Again, we'd usually know the rough timeframe, say +/- a few days.

3. Looking at all requests that hit a particular endpoint, basically filtering on 2-3 log entry properties.

Could these be supported by regexp filtering, grep style? Perhaps, but it would depend on how quick that filtering/lookup would be.

Some stats:

• Storing 60 days of log data
• 26M log lines (with ~10 fields of metadata per line)
• 130 gb total
• 2-3 gb per day

Since our log volumes are so small, maybe it'll be easy to grep through?

---

Regarding your "jq style language" suggestion, I think that's a great idea! Even better would be UI support for key-value filtering on keys in the top-level of that json document. Usually people will have the log message as one key (for example message), and the metadata as other top-level keys.

Logging JSON lines seems to be common enough that it's worth building some UI/support around it:

• https://www.loggly.com/blog/why-json-is-the-best-application-log-format-and-how-to-switch/
• https://docs.fluentd.org/v0.12/articles/parser_json
• https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#format-dictionaries
• https://docs.docker.com/config/containers/logging/json-file/
• https://jahed.io/2018/07/05/always-log-to-json/

---

I've copied parts of this comment into the top-level issue, for clarity. Still kept here too, for context/history.

[mumoshu]

Would this make sense in regard to storing/searching traces and spans, too?

I think I saw that Grafana had recently announced the "LGTM" stack, where T stands for Trace/Tracing. My "impression" on the announcement was that, you may be going to use Loki with optional indices on, say "trace id" and "span id", to make traces stored in Loki searchable, so that it can be an alternative datastore for Zipkin or Jaeger.

I don't have a strong opinion if this should be in Loki or not. Just wanted to discuss and get the idea where we should head in relation to the LGTM stack 😃

[mumoshu]

Never mind on my previous comment.

I think we don't need to force Loki to blur its scope and break its project goals here.

We already have other options like Elasticsearch(obviously), Badger, RocksDB, BoltDB, and so on when it comes to a trace/span storage, as seen in Jaeger https://github.com/jaegertracing/jaeger/pull/760, for example.

Regarding distributed logging, nothing prevents us from implementing our own solution with promtail + badger for example, or using Loki in combination with an another distributed logging solution w/ a more richer indexing.

I'd just use Loki for light-weight, short-term, cluster-local distributed logging solution. I use Prometheus with a similar purpose for metrics. Loki remains super useful even without high cardinality labels support or secondary indices. Just my two cents 😃

[candlerb]

I've made a suggestion in point (2) here with a simple way of dealing with high cardinality values.

In short: keep the high-cardinality fields out of the label set, but parse them as pseudo-labels which are regenerated on demand when reading the logs.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[sandstrom]

Stale bot: I don't think it makes sense to close this issue.

Although I understand that high cardinality labels may not be on the immediate roadmap, it's a common feature of log aggregation systems and by keeping it open there will at least be a place where this + workarounds can be discussed.

[candlerb]

An alternative to high cardinality labels could be to introduce a complement to labels called metadata, that is allowed to be high cardinality. Having two different things could allow us to impose other restrictions on the metadata type of values.

If grep is quick enough, maybe it would work to not index the metadata key/value pairs. But still allow them to be filtered (at "grep speed"). That would allow you to give these high-cardinality key-value pairs blessed UI, for easy filtering, whilst still avoiding the cost of indexing high cardinality fields.

This approach is very similar to influxdb, which has tags and fields. Influxdb tags are like loki labels: each set of unique tags defines a time series. Fields are just stored values.

Even linear searching would be "better than grep" speed, since if the query contains any labels, loki would already filter down to the relevant chunks.

Idea 1

Store arbitrary metadata alongside the line.

In order not to complicate the query language, I suggest that these stored-but-unindexed metadata fields look like labels with a special format: e.g. they start with a special symbol (e.g. _). Then you could do:

query {router="1.2.3.4", type="netflow", _src_ip=~"10\.10\.1\..*"} foo

In terms of the API, submitting records could be done the same way. Unfortunately, it would defeat the grouping of records with same label set, making the API more verbose and making it harder for loki to group together records belonging to the same chunk. Example:

{
  "streams": [
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\",_src_ip=\"10.10.1.50\"}",
      "entries": [{ "ts": "2018-12-18T08:28:06.801064-04:00", "line": "...netflow record 1..." }]
    },
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\",_src_ip=\"10.10.1.84\"}",
      "entries": [{ "ts": "2018-12-18T08:28:06.801065-04:00", "line": "...netflow record 2..." }]
    },
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\",_src_ip=\"10.10.1.42\"}",
      "entries": [{ "ts": "2018-12-18T08:28:06.801066-04:00", "line": "...netflow record 3..." }]
    }
  ]
}

So in the API it might be better to include the metadata inline in each record:

{
  "streams": [
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\"}",
      "entries": [
        { "ts": "2018-12-18T08:28:06.801064-04:00", "_src_ip": "10.10.1.50", "line": "...netflow record 1..." },
        { "ts": "2018-12-18T08:28:06.801065-04:00", "_src_ip": "10.10.1.84", "line": "...netflow record 2..." },
        { "ts": "2018-12-18T08:28:06.801066-04:00", "_src_ip": "10.10.1.42", "line": "...netflow record 3..." }
      ]
    }
  ]
}

In the grafana UI you could have one switch for displaying labels, and one for displaying metadata.

Wild idea 2

The same benefits as the above could be achieved if the "line" were itself a JSON record: stored in native JSON, and parsed and queried dynamically like jq.

You can do this today, if you just write serialized JSON into loki as a string. Note: this sits very well with the ELK/Beats way of doing things.

{
  "streams": [
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\"}",
      "entries": [
        { "ts": "2018-12-18T08:28:06.801064-04:00", "line":"{\"_src_ip\": \"10.10.1.50\", \"record\": \"...netflow record 1...\"}" },
        { "ts": "2018-12-18T08:28:06.801065-04:00", "line":"{\"_src_ip\": \"10.10.1.84\", \"record\": \"...netflow record 2...\"}" },
        { "ts": "2018-12-18T08:28:06.801066-04:00", "line":"{\"_src_ip\": \"10.10.1.42\", \"record\": \"...netflow record 3...\"}" }
      ]
    }
  ]
}

The only thing you can't do today is the server-side filtering of queries.

Therefore, you'd need to extend the query language to allow querying on fields of the JSON "line", in the main query part outside the label set. I think that using jq's language could be the ideal solution: the world does not need another query language, and it would allow querying down to arbitrary levels of nesting.

As an incremental improvement, you could then extend the API to natively accept a JSON object, handling the serialization internally:

{
  "streams": [
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\"}",
      "entries": [
        { "ts": "2018-12-18T08:28:06.801064-04:00", "line":{"_src_ip": "10.10.1.50", "record": "...netflow record 1..."} },
        { "ts": "2018-12-18T08:28:06.801065-04:00", "line":{"_src_ip": "10.10.1.84", "record": "...netflow record 2..."} },
        { "ts": "2018-12-18T08:28:06.801066-04:00", "line":{"_src_ip": "10.10.1.42", "record": "...netflow record 3..."} }
      ]
    }
  ]
}

On reading records back out via the API, it could be optional whether loki returns you the raw line inside a string, or (for valid JSON) gives you the object. There are certainly cases where you'd want to retain the string version, e.g. for logcli.

Storing a 1-bit flag against each line, saying whether it's valid JSON or not, could be a useful optimisation.

The big benefits of this approach are (a) no changes to underlying data model; (b) you gain much more power dealing with log data which is already natively JSON; (c) you gain the ability to filter on numeric fields, with numeric comparison operators like < and > (where 2 < 10, unlike "2" > "10")

---

Aside: for querying IP address ranges, regexps aren't great. One option would be to enhance the query language with IP-address specific operators. Another one is to convert IP addresses to fixed-width hexadecimal (e.g. 10.0.0.1 = "0a000001") before storing them in metadata fields; prefix operations can be converted to regexp mechanically.

e.g. 10.0.0.0/22 => 0a000[0123].* or better 0a000[0123][0-9a-f]{2}

[kiney]

I like both ideas, especially the second one. For the query language i think JMESPath would make more sense than the jq language as it is already supported for log processing in scrape config: https://github.com/grafana/loki/blob/master/docs/logentry/processing-log-lines.md

I though about extracting some Info there and setting it as label. What would actually happen when creating high cardinality labels that way? "just" a very big index or would I run into real problems?

[candlerb]

What would actually happen when creating high cardinality labels that way? "just" a very big index or would I run into real problems?

You could end up with Loki trying to create, for example, 4 billion different timeseries, each containing one log line. Since a certain amount of metadata is required for each timeseries, and (I believe) each timeseries has its own chunks, it is likely to explode.

Also, trying to read back your logs will be incredibly inefficient, since it will have to merge those 4 billion timeseries back together for almost every query.

[candlerb]

@kiney: I hadn't come across JMESPath before, and I agree it would make sense to use the engine which is already being used. We need something that can be used as a predicate to filter a stream of JSON records, and ideally with a CLI for testing.

It looks like JMESPath's jp is the JMESPath equivalent to jq. (NOTE: this is different to the JSON Plotter jp which is what "brew install jp" gives you)

jp doesn't seem to be able to filter a stream of JSON objects like jq does. It just takes the first JSON item from stdin and ignores everything else.

As a workaround, loki could read batches of lines into JSON lists [ ... ] and apply JMESPath to each batch. Then you could use JMESPath list filter projections, e.g.

[?foo=='bar']

It needs to work in batches to avoid reading the whole stream into RAM: therefore functions which operate over all records, like max_by, would not work. But to be fair, I don't think jq can do this either, without the -s (slurp into list) option, or unless you write your own functions.

I think the most common use case for logs is to filter them and return the entire record for each one selected. It would be interesting to make a list of the most common predicates and see how they are written in both jq and jp. e.g.

• Select records containing top-level element "foo"
• Select records where top-level element "foo" is string "bar"
• Select records where top-level element "foo" is string "bar" and element "baz" is "qux"
• Select records where top-level element "foo" matches regexp ^ba[rz]$ (*)
• Select records where top-level numeric element "baz" is greater than 200 ... etc

(*) Note: regular expressions are supported by jq but not as far as I can tell by JMESPath

Just taking the last example:

jq 'select(.baz > 200)' <<EOS
{"foo":"bar"}
{"foo":"baz"}
{"foo":"bar","baz":123}
{"foo":"bar","baz":789}
{"foo":"baz","baz":456}
EOS

=>
{
  "foo": "bar",
  "baz": 789
}
{
  "foo": "baz",
  "baz": 456
}

Compare:

jp '[?baz > `200`]' <<EOS
[{"foo":"bar"},
{"foo":"baz"},
{"foo":"bar","baz":123},
{"foo":"bar","baz":789},
{"foo":"baz","baz":456}]
EOS

=>
[
  {
    "baz": 789,
    "foo": "bar"
  },
  {
    "baz": 456,
    "foo": "baz"
  }
]

Note the non-obvious requirement for backticks around literals. If you omit them, the expression fails:

jp '[?baz > 200]' <<EOS
[{"foo":"bar"},
{"foo":"baz"},
{"foo":"bar","baz":123},
{"foo":"bar","baz":789},
{"foo":"baz","baz":456}]
EOS

=>
SyntaxError: Invalid token: tNumber
[?baz > 200]
        ^

String literals can either be written in single quotes, or wrapped in backticks and double-quotes. I find it pretty ugly, but then again, so are complex jq queries.

Aside: If you decide to give jq a set of records as a JSON list like we just did with jp, then it's a bit more awkward:

jq 'map(select(.baz > 200))' <<EOS  # or use -s to slurp into a list
[{"foo":"bar"},
{"foo":"baz"},
{"foo":"bar","baz":123},
{"foo":"bar","baz":789},
{"foo":"baz","baz":456}]
EOS

=>
[
  {
    "foo": "bar",
    "baz": 789
  },
  {
    "foo": "baz",
    "baz": 456
  }
]

But you probably wouldn't do that anyway with jq.

[candlerb]

There's another way jq could be used: just for predicate testing.

jq '.baz > 200' <<EOS
{"foo":"bar"}
{"foo":"baz"}
{"foo":"bar","baz":123}
{"foo":"bar","baz":789}
{"foo":"baz","baz":456}
EOS

=>
false
false
false
true
true

If loki sees the value 'false' it could drop the record, and the value 'true' it passes the original record through unchanged. Any other value would be passed through. This would allow the most common queries to be simplified.

[kiney]

@candlerb I also consider jq a slightly more pleasant language but JMESPath is already used in Loki and also the standard in other cloud tooling (aws cli, azure cli, ansible...). I thought about using it similiar to current regex support: apply on each line, filter out lines with no match and highlight matched part in the remaining lines.

[sandstrom]

Updates

• This design proposal doesn't directly address high-cardinality labels, but may solve the same underlying problem.

  For some use-cases it would be equal to high-cardinality labels, as long as the LogQL queries can run fast enough on large log

• There is some discussion in this thread, on making it easier to find unique IDs, such as trace-id and request-id

EDIT: copied to top comment

[feldentm-SAP]

This also affects handling of OpenTelemetry attributes and TraceContext especially if the set of attributes is not trivial. If you have thousands of service instances in your Loki and you want to do queries like "give me all lines that have a certain action attribute set and failed" you cannot restrict that query on instances. Restricting it by time is also an issue because you might miss some interesting events. Not having it is an issue in detecting root causes of sporadic defects.

[lsampras]

Are there any further updates on this issue..

I see some solutions being built for trace-id use-case...

But is there a plan for supporting user-defined high-cardinality labels over a large time span...

[candlerb]

I think you can do what you need using the LogQL json filter. This extracts "labels" from the JSON body, but they are not logstream labels, so they don't affect cardinality.

These are not indexed, so it will perform a linear search across the whole timespan of interest. If you scale up Loki then it might be able to parallelize this across chunks - I'm not sure.

But if you want fully indexed searching, then I believe Loki isn't for you. You can either:

• throw a ton of resources at the problem with Elasticsearch (tons of RAM, tons of SSD, expect your log volumes to increase by a factor of 10); or
• dump all your logs into one of the "big data" products (hive, druid, kudu, drill, clickhouse etc), possibly using an on-disk format like parquet optimised for columnar searching; or
• dip your toe in the water with the recently-announced VictoriaLogs. This looks like it hits a sweet spot with word splitting and indexing, but is not yet considered production-ready. Maybe Loki will consider adding something similar in future.

[sandstrom]

For others interested in this, this is the most recent "statement" I've seen from the Loki team:

https://github.com/grafana/loki/issues/1282#issuecomment-573678616

[lsampras]

But if you want fully indexed searching, then I believe Loki isn't for you.

Yeah, Creating a complete index would be out-of-scope and against the design of Loki

But are we considering some middle ground for e.g

1. the ability to define certain labels for which chunks would be bloom filtered instead of separate streams
2. letting users specify chunk id's in the loki query request so that they can build/maintain their own indexing etc.
3. we've also had some ideas around adding a co-processor

I wanted to gauge what in this case would we be willing to extend in loki to support this (without going against its core design principles)

[valyala]

Loki can implement support for high-cardinality labels in the way similar to VictoriaLogs - it just doesn't associate these labels with log streams. Instead, it stores them inside the stream - see storage architecture details. This allows resolving the following issues:

• High RAM usage, OOM crashes, high number of open files, significant slowdown
• Slow query performance when high-cardinality labels are stored in the log message itself.

The high-cardinality labels can then be queried with label_name:value filters - see these docs for details.

[sandstrom]

Time flies!

Issue number 91 might finally have a nice solution! 🎉

I don't want to say never, but I'm not sure if this is something we're ever likely to support.

@tomwilkie I'm happy it wasn't never 😄

Six years later, there is a solution on the horizon:

• https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata
• https://grafana.com/blog/2023/12/20/the-concise-guide-to-grafana-loki-everything-you-need-to-know-about-labels/#i-want-to-search-all-my-logs-for-a-unique-id

(not GA just yet though)

[feldentm-SAP]

Hi @sandstrom,

please inform us and your support organization, once this is GA. Also, please make sure that the GA version will allow ingesting data that has the structured-metadata already irrespective of the kind of log source. For OTLP, there should be means of configuring what to keep or transforming the log into whatever your internal format is going to be.

Kind regards and thanks for the great contribution

[sandstrom]

A quick update for people following this. Loki 3.0 has shipped with experimental bloom filters, that are addressing exactly this issue. This is awesome news! 🎉

https://grafana.com/blog/2024/04/09/grafana-loki-3.0-release-all-the-new-features/

---

I'll keep this open until it's no longer experimental, which is probably in 3.1 I'd guess, unless a team member of Loki wants to close this out already, since it has shipped (albeit in an experimental state) -- I'm fine with either.

[chaudum]

I'll keep this open until it's no longer experimental, which is probably in 3.1 I'd guess, unless a team member of Loki wants to close this out already, since it has shipped (albeit in an experimental state) -- I'm fine with either.

Sounds good to me :smile: