p1-repo-sync-bot[bot]
commented
4 months ago jimmyungerman commented:
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
Github Comment Mirrored Here
After reading the Best Practices doc, it appears we want the absolute least amount of labels as possible so we don't have to chunk our queries.
To do this, I'm formally proposing the following default labels config:
pipelineStages:
- cri: {}
- labeldrop:
- filename
common:
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node_name
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- action: replace
replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
scrapeConfigs: |
- job_name: kubernetes-pods
pipeline_stages:
{{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
kubernetes_sd_configs:
- role: pod
relabel_configs:
{{- if .Values.config.snippets.addScrapeJobLabel }}
- replacement: kubernetes-pods
target_label: scrape_job
{{- end }}
{{- toYaml .Values.config.snippets.common | nindent 4 }}
{{- with .Values.config.snippets.extraRelabelConfigs }}
{{- toYaml . | nindent 4 }}
{{- end }}This supplies us with the following default labels for all pods:
And shows we're drastically reducing our chunk flushing as well:
This should greatly improve the speed of our Loki queries.
@andrewshoell @chris.oconnell @michaelmartin Any thoughts?
Bug
Description
When deploying PLG to any sizable K8s cluster with a normal use of K8s labels, the Promtail scrapeconfig relabel configs are causing Loki to store logs in chunks that are too small to retrieve efficiently since Loki stores logs by unique combinations of labels (see label best practices in the Loki docs). While this means log collection works fine, querying logs can take a very long time and timeouts have to be continuously extended. In order to resolve this, the entire default scrapeconfig string has to be overridden to remove all the extra, unneeded labels.
Could the actions here and here in the scrapeconfig either be removed or moved under extra scrapeconfigs so that deployments to sizable clusters can get reasonably sized chunks and timely query results more easily?
BigBang Version
What version of BigBang were you running? Any