When deploying PLG to any sizable K8s cluster with a normal use of K8s labels, the Promtail scrapeconfig relabel configs are causing Loki to store logs in chunks that are too small to retrieve efficiently since Loki stores logs by unique combinations of labels (see label best practices in the Loki docs). While this means log collection works fine, querying logs can take a very long time and timeouts have to be continuously extended. In order to resolve this, the entire default scrapeconfig string has to be overridden to remove all the extra, unneeded labels.
Could the actions here and here in the scrapeconfig either be removed or moved under extra scrapeconfigs so that deployments to sizable clusters can get reasonably sized chunks and timely query results more easily?
Bug
Description
When deploying PLG to any sizable K8s cluster with a normal use of K8s labels, the Promtail scrapeconfig relabel configs are causing Loki to store logs in chunks that are too small to retrieve efficiently since Loki stores logs by unique combinations of labels (see label best practices in the Loki docs). While this means log collection works fine, querying logs can take a very long time and timeouts have to be continuously extended. In order to resolve this, the entire default scrapeconfig string has to be overridden to remove all the extra, unneeded labels.
Could the actions here and here in the scrapeconfig either be removed or moved under extra scrapeconfigs so that deployments to sizable clusters can get reasonably sized chunks and timely query results more easily?
BigBang Version
What version of BigBang were you running? Any