[Blueprints, 02-at-scale]: Make enablement of the Grafana Ingress compatible with terraform destroy

[carlosrodlop]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

Currently Grafana is accesible via URL instead of using kube-proxy.

Then you connect define it as OpenTelemetry Backend ${sec_grafanaUrl} in the following casc snippet for parent controller

main.tf

  grafana_hostname = "grafana.${var.hosted_zone}"
  grafana_url      = "https://${local.grafana_hostname}"

kube-prom-stack-values.yml

  openTelemetry:
    endpoint: ${ot_endpoint}
    observabilityBackends:
    - grafana:
        grafanaBaseUrl: ${sec_grafanaUrl}
        tempoDataSourceIdentifier: "Tempo"

Describe the solution you would like

The following configuration for grafana enables ingress correctly

kube-prom-stack-values.yml

grafana:
...
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
      alb.ingress.kubernetes.io/certificate-arn: ${cert_arn}
      alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
      external-dns.alpha.kubernetes.io/hostname: ${grafana_hostname}
    hosts:
      - ${grafana_hostname}

but there is a problem ==> the created ELB is not wiped out during terraform destroy. Even I created a separated namespace for kube-prometheus-stack from the terrafom eks blueprints

Possible solution

Run a script via terraform_data > local exec to delete load balancer based on tags-values e.g. ingress.k8s.aws/stack - kube-prometheus-stack/kube-prometheus-stack-grafana

#!/bin/bash

# Desired tag key and value
TAG_KEY="YourTagKey"
TAG_VALUE="YourTagValue"

# List all ALBs
load_balancers=$(aws elbv2 describe-load-balancers --query 'LoadBalancers[*].LoadBalancerArn' --output text)

# Loop through all load balancers to find the one with the desired tag
for lb_arn in $load_balancers; do
    # Describe tags for the current load balancer
    tags=$(aws elbv2 describe-tags --resource-arns $lb_arn)
    # Check if the desired tag is present
    if echo $tags | grep -q "\"Key\": \"$TAG_KEY\", \"Value\": \"$TAG_VALUE\""; then
        echo "Deleting Load Balancer with ARN: $lb_arn"
        # Delete the load balancer
        aws elbv2 delete-load-balancer --load-balancer-arn $lb_arn
        break # Assuming only one load balancer matches; remove this if there could be multiple
    fi
done

Additional context