meta-mimir-nginx-XYZ CrashLoopBackOff due to IPv6 configuration issue

[fara-moeeni]

Hello, I am experiencing an issue with the pod meta-mimir-nginx-8fd474c8d-9znsz, which is in a CrashLoopBackOff state. Below are the details:

Pod Status: meta-mimir-nginx-8fd474c8d-9znsz 0/1 CrashLoopBackOff 36 (3m9s ago) 163m

Pod Logs: /docker-entrypoint.sh: No files found in /docker-entrypoint.d/, skipping configuration 2024/08/25 18:48:49 [emerg] 1#1: socket() [::]:8080 failed (97: Address family not supported by protocol) nginx: [emerg] socket() [::]:8080 failed (97: Address family not supported by protocol)

Configuration: I have added the following configuration(gateway.nginx.config) in values.yaml: mimir-distributed: gateway: nginx: config: enableIPv6: false minio: enabled: false global: extraEnvFrom:

• secretRef: name: "minio" dnsNamespace: "kube-system" dnsService: "rke2-coredns-rke2-coredns" nginx: config: enableIPv6: false

It is worth mentioning that the meta-loki-gateway also had the same problem. I looked into the Loki Helm chart and added the configuration to the Nginx part of the Gateway, setting enableIPv6 to false. This worked for the Loki Gateway, but I couldn’t fix it for the Mimir Nginx.

Despite this configuration, the issue persists. Could you please provide guidance on how to resolve this issue? Thank you

[tchabit]

Hello @fara-moeeni,

I've had the same issue and it was due to a missing variable definition/use on the mimir-distributed values.yaml file, used as template to create the nginx config. https://github.com/grafana/mimir/blob/mimir-distributed-5.3.0/operations/helm/charts/mimir-distributed/values.yaml

The 'enableIPv6' variable is set one time (gateway.nginx.config.enableIPv6 - line 2949), and used on gateway.nginx.config.file (line 3010) :

server {
  listen {{ include "mimir.serverHttpListenPort" . }};
  {{- if .Values.gateway.nginx.config.enableIPv6 }}
  listen [::]:{{ include "mimir.serverHttpListenPort" . }};
  {{- end }}

But it appears that the config file used to create the configmap is the one defined on nginx.nginxConfig.file (line 2615), where there is no variable use :

server {
  listen 8080;
  listen [::]:8080;

By editing/patching the configmap of the nginx pod to delete the IPv6 listen config, it should work.

[bimax]

I had the same issue in my k8s cluster when updated loki some time ago (our cluster does not support IPv6 now). Configuration which helped me to exclude listen [::]:8080; is as follow:

nginxConfig:
  file: |
    {{- include "loki.nginxFile" . | replace "listen             [::]:8080;" "" | indent 2 -}}