search

prometheus-community / helm-charts

Prometheus community Helm charts
Apache License 2.0
4.86k stars 4.93k forks source link

[kube-prometheus-stack] I am not able to create a prometheus ingress with aws alb controller. #4403

Open a-kumar5 opened 3 months ago

a-kumar5 commented 3 months ago

Describe the bug a clear and concise description of what the bug is.

I have enabled the ingress and provided all the necesaary information for annotations and hosts. The load balancer with target is getting created but the healthcheck is getting failed and we are not able to access the prometheus ui.

What's your helm version?

version.BuildInfo{Version:"v3.13.3", GitCommit:"c8b948945e52abba22ff885446a1486cb5fd3474", GitTreeState:"clean", GoVersion:"go1.20.11"

What's your kubectl version?

Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.10-eks-508b6b3

Which chart?

kube-prometheus-stack

What's the chart version?

57.1.1

What happened?

The alb helath check is getting failed.

What you expected to happen?

The alb healthcheck should pass and i should be able to access the prometheus ui.

How to reproduce it?

Create a ingress with stack using the aws alb controller and the healthcheck will fail.

Enter the changed values of values.yaml?

prometheus: prometheusSpec: podMonitorSelectorNilUsesHelmValues: false serviceMonitorSelectorNilUsesHelmValues: false externalUrl: "http://prometheus-dashboard.stage.nextgen.shs.com/prometheus" routePrefix: /

ingress: enabled: true annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/group.name: 'monitoring.prometheus-dashboard-internal-ingress' alb.ingress.kubernetes.io/group.order: '1' alb.ingress.kubernetes.io/scheme: 'internal' alb.ingress.kubernetes.io/security-groups: sg-056d4775173488c2b external-dns.alpha.kubernetes.io/hostname: prometheus-dashboard.stage.nextgen.oh.shs.com external-dns.alpha.kubernetes.io/route53-zone-type: private alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]' alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:523140806637:certificate/f1202f91-8b24-4c19-b361-de22c3ad5739 alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06 alb.ingress.kubernetes.io/healthcheck-port: traffic-port alb.ingress.kubernetes.io/healthcheck-protocol: HTTP alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '10' alb.ingress.kubernetes.io/healthcheck-interval-seconds: '60' alb.ingress.kubernetes.io/healthy-threshold-count: '2' alb.ingress.kubernetes.io/unhealthy-threshold-count: '10' alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=60 alb.ingress.kubernetes.io/success-codes: '200-299' labels: k8s-app: prometheus-dashboard

hosts:
  - prometheus-dashboard.stage.nextgen.shs.com

## Paths to use for ingress rules - one path should match the prometheusSpec.routePrefix
##
paths:
  - /prometheus(/|$)(.*)

Enter the command that you execute and failing/misfunctioning.

helm upgrade prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --namespace monitoring --values values.yml

Anything else we need to know?

My eks is in private network.

jimbeck commented 3 months ago

I'm running into this issue too. I have grafana and the alertmanager ingress working fine using the alb ingress controller. For some reason with prometheus it doesn't create the target group in the load balancer. Will report back if I figure it out!