Cardinality too high due to `filename` label in some use-cases

[facundofc]

Enhancement Proposal

When using Loki to ingest logs from an Openstack cloud, the inclusion of the filename label for each log line can potentially lead to a huge cardinality.

Openstack uses libvirt to spawn VMs, and libvirt creates one log file for each VM launched (/var/log/libvirt/qemu/$DOMAIN_NAME.log). Some clouds used as build farms, can spawn thousands of VMs in the course of a few days, each with it's own value for the filename label.

We need a way to customize Loki so as to avoid including the filename label in some files (or directories?). To not lose querying capabilities (as these files don't have the domain name on each line), maybe we could consider using structured metadata?

[simskij]

@sed-i we should probably treat this is a high/critical priority item for the coming pulse.

[Abuelodelanada]

We should allow the user to override the default "/var/log/**/*log" or the entire scrape job.

[facundofc]

FTR and given @Abuelodelanada's comment, we need these logs. So not scraping them is not an option.

[lucabello]

Overriding the default scrape job doesn't mean "not scraping those logs"; you can override the default with the same targets, but define custom labels: this is how you would get the same logs, but without the automatic label injection for the filename label.

If I understand correctly, that's the result you want to achieve, or am I missing something?

[facundofc]

Hey sorry for the delayed reply.

Yes, your understanding is correct. I was just trying to make it explicit to eliminate all doubt (as the "or the entire scrape job" could be interpreted as providing a toggle to either enable or disable the scrapping).