search

grafana / helm-charts

Apache License 2.0
1.66k stars 2.27k forks source link

[loki-stack] Promtail unable to push logs to Loki, because the url may be wrong. #2102

Open cero-t opened 1 year ago

cero-t commented 1 year ago

After installing loki-stack, Promtail causes "no such host" error when pushing logs to Loki.

I found the reason of this problem. When the release name contains "loki", the Loki's service name will be the same as the release name, like "my-loki". Otherwise the suffix "-loki" will be given, like "monitoring-loki" with its release name "monitoring". However, promtail.config.clients.url in values.yml is now {{ .Release.Name }}:3100/loki/api/v1/push, which doesn't care the suffix nad causes an error if the service name does not contain "loki".

(from values.yaml)

promtail:
  config:
    clients:
      - url: http://{{ .Release.Name }}:3100/loki/api/v1/push

To solve this problem, I think promtail.config.clients.url should be {{ .Values.loki.url }}/loki/api/v1/push.

And I found same problem in the Logstash configuration

  outputs:
    main: |-
      output {
        loki {
          url => "http://loki:3100/loki/api/v1/push"

I didn't try with Logstash, but I guess it causes the same problem.

batazor commented 1 year ago

any update

LucasPAmaral commented 9 months ago

I've tracked down this problem, there is a _helpers.tpl defining loki.singleBinaryFullname in the loki charts. It does pretty-much the same as what the loki.fullname does as well. It contains the default behaviour of a freshly created helm charts. Assuming you do a helm install without a fullnameOverride or enterprise enabled, there are two possible outcomes: 1- Using a release name containing loki -> loki.singleBinaryFullname will look like the exact release name provided, {{ .Release.Name }}. 2- Using a release name not containig loki -> loki.singleBinaryFullname will look like {{ .Release.Name }}-{{ .Chart.Name }}, later translated to releasename-loki. So, when configuring the promtail client url variable in the charts, there will be a mismatch to the actual service name when the release name doesn't contains the loki substring. A possibility of a fix it is to create a variable in the _helpers.tpl to handle that in the promtail charts and include that variable to the default loki-stack values, instead of {{ .Release.Name }}.

P.S.: There seems to be a sollution with the variable loki.servicename in the loki-stack charts. P.S.2: There once was a variable loki.servicename th the promtail charts, but was removed here.