Helm chart: How to enable HTTPS connection

[andormarkus]

Apache Airflow version: 2.0.2 Helm chart version: 1.0.0 Kubernetes version: 1.20

What happened: I have tried to enable HTTPS connection, however when I enable it liveness and readiness probes are failing

▶ kubectl -n airflow describe pod/airflow-webserver-564fcf9c5c-jqtt4

Name:         airflow-webserver-564fcf9c5c-jqtt4
Namespace:    airflow
Priority:     0
Node:         ip-10-10-13-31.eu-central-1.compute.internal/10.10.13.31
Start Time:   Sun, 23 May 2021 11:37:08 +0200
Labels:       component=webserver
              pod-template-hash=564fcf9c5c
              release=airflow
              tier=airflow
Annotations:  checksum/airflow-config: 4e5f26f48f157218dcaf8eddac95e912aa7a47a4eb75b04697bb3701fd550340
              checksum/extra-configmaps: 2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
              checksum/extra-secrets: bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
              checksum/metadata-secret: 765446a9def21895960ebc0df295399c35d3467cdd84be6ef8bc65c18ca0f7e5
              checksum/pgbouncer-config-secret: da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
              checksum/webserver-config: 4a2281a4e3ed0cc5e89f07aba3c1bb314ea51c17cb5d2b41e9b045054a6b5c72
              kubernetes.io/psp: eks.privileged
Status:       Running
IP:           10.10.13.56
IPs:
  IP:           10.10.13.56
Controlled By:  ReplicaSet/airflow-webserver-564fcf9c5c
Init Containers:
  wait-for-airflow-migrations:
    Container ID:  docker://da6e1a36ac496982bc6d9058e1d94791cf99bf7d70dbbe10be2f2bbe0446eae8
    Image:         apache/airflow:2.0.2
    Image ID:      docker-pullable://apache/airflow@sha256:1c3dbd1c3e964e98fffcb58efb77b41179dc1322a9d3919e6f4289f2d2d84625
    Port:          <none>
    Host Port:     <none>
    Args:
      python
      -c
      import airflow
      import logging
      import os
      import time

      from alembic.config import Config
      from alembic.runtime.migration import MigrationContext
      from alembic.script import ScriptDirectory

      from airflow import settings

      package_dir = os.path.abspath(os.path.dirname(airflow.__file__))
      directory = os.path.join(package_dir, 'migrations')
      config = Config(os.path.join(package_dir, 'alembic.ini'))
      config.set_main_option('script_location', directory)
      config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN.replace('%', '%%'))
      script_ = ScriptDirectory.from_config(config)

      timeout=60

      with settings.engine.connect() as connection:
          context = MigrationContext.configure(connection)
          ticker = 0
          while True:
              source_heads = set(script_.get_heads())

              db_heads = set(context.get_current_heads())
              if source_heads == db_heads:
                  break

              if ticker >= timeout:
                  raise TimeoutError("There are still unapplied migrations after {} seconds.".format(ticker))
              ticker += 1
              time.sleep(1)
              logging.info('Waiting for migrations... %s second(s)', ticker)

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 23 May 2021 11:37:11 +0200
      Finished:     Sun, 23 May 2021 11:37:24 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      AIRFLOW__CORE__FERNET_KEY:        <set to the key 'fernet-key' in secret 'airflow-fernet-key'>         Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-postgres-password'>  Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:          <set to the key 'connection' in secret 'airflow-postgres-password'>  Optional: false
      AWS_DEFAULT_REGION:               eu-central-1
      AWS_REGION:                       eu-central-1
      AWS_ROLE_ARN:                     arn:aws:iam::589348632636:role/dataeng-iamrole-eksairflow-dev
      AWS_WEB_IDENTITY_TOKEN_FILE:      /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from airflow-token-9cqzb (ro)
Containers:
  webserver:
    Container ID:  docker://2b8ee8a99ebd088b545685b3d1bc290adb0a7675e4935322a9549a152339a249
    Image:         apache/airflow:2.0.2
    Image ID:      docker-pullable://apache/airflow@sha256:1c3dbd1c3e964e98fffcb58efb77b41179dc1322a9d3919e6f4289f2d2d84625
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      bash
      -c
      exec airflow webserver
    State:          Running
      Started:      Sun, 23 May 2021 11:39:48 +0200
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 23 May 2021 11:37:26 +0200
      Finished:     Sun, 23 May 2021 11:39:48 +0200
    Ready:          False
    Restart Count:  1
    Liveness:       http-get http://:8080/airflow/health delay=15s timeout=30s period=5s #success=1 #failure=20
    Readiness:      http-get http://:8080/airflow/health delay=15s timeout=30s period=5s #success=1 #failure=20
    Environment:
      AIRFLOW__CORE__FERNET_KEY:        <set to the key 'fernet-key' in secret 'airflow-fernet-key'>         Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-postgres-password'>  Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:          <set to the key 'connection' in secret 'airflow-postgres-password'>  Optional: false
      AWS_DEFAULT_REGION:               eu-central-1
      AWS_REGION:                       eu-central-1
      AWS_ROLE_ARN:                     arn:aws:iam::589348632636:role/dataeng-iamrole-eksairflow-dev
      AWS_WEB_IDENTITY_TOKEN_FILE:      /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
      /opt/airflow/dags from dags (rw)
      /opt/airflow/logs from logs (rw)
      /opt/airflow/secrets/ssl from airflow-ssl (ro)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from airflow-token-9cqzb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      airflow-airflow-config
    Optional:  false
  dags:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  airflow-dags
    ReadOnly:   false
  logs:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  airflow-logs
    ReadOnly:   false
  airflow-ssl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-ssl
    Optional:    false
  airflow-token-9cqzb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-token-9cqzb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  namespace=airflow
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m14s                  default-scheduler  Successfully assigned airflow/airflow-webserver-564fcf9c5c-jqtt4 to ip-10-10-13-31.eu-central-1.compute.internal
  Normal   Pulled     4m11s                  kubelet            Container image "apache/airflow:2.0.2" already present on machine
  Normal   Created    4m11s                  kubelet            Created container wait-for-airflow-migrations
  Normal   Started    4m11s                  kubelet            Started container wait-for-airflow-migrations
  Normal   Pulled     3m57s                  kubelet            Container image "apache/airflow:2.0.2" already present on machine
  Normal   Created    3m57s                  kubelet            Created container webserver
  Normal   Started    3m56s                  kubelet            Started container webserver
  Warning  Unhealthy  3m37s                  kubelet            Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": dial tcp 10.10.13.56:8080: connect: connection refused
  Warning  Unhealthy  3m36s (x2 over 3m41s)  kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": dial tcp 10.10.13.56:8080: connect: connection refused
  Warning  Unhealthy  3m2s                   kubelet            Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  3m1s                   kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:37846->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  3m1s                   kubelet            Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38016->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  3m1s                   kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38046->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  3m1s                   kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38048->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  2m57s                  kubelet            Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38056->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  2m56s                  kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38060->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  2m32s (x9 over 2m52s)  kubelet            (combined from similar events): Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38192->10.10.13.56:8080: read: connection reset by peer

How to reproduce it: Helm chart myconf.yaml looks like this

executor: KubernetesExecutor
fernetKey: "XXXXXXXXXX"

webserver:
  base_url: "https://k8s.foobaa.com/airflow"
  web_server_ssl_cert: "/opt/airflow/secrets/ssl/tls.crt"
  web_server_ssl_key: "/opt/airflow/secrets/ssl/tls.key"

webserver:
  service:
    type: NodePort
  extraVolumeMounts:
    - name: airflow-ssl
      mountPath: /opt/airflow/secrets/ssl
      readOnly: true
  extraVolumes:
    - name: airflow-ssl
      secret:
        secretName: airflow-ssl

[potiuk]

I believe all the deployments in K8S should be done using http (or if istio is enabled they can be additionally protected via mTLS).

I think the usual pattern of providing SSL deployment is via external proxy/gateways, especially that they can then provide additional authentication mechanism (Oauth/client certificates etc.) which can be configured outside of Airflow and even provide single authentication/SSL endpoints for multiple services.

I think this is common best practice to decouple SSL termination and the service itself.

Here is a example of simple nginx-based SSL proxy that you can follow: https://kubernetes.io/blog/2015/07/strong-simple-ssl-for-kubernetes/

[mik-laj]

You just don't need to configure Nginx, but you can configure SSL in Ingress. This will also allow certmanager to automatically manage the certificates.

Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/cluster-issuer: nameOfClusterIssuer
  name: myIngress
  namespace: myIngress
spec:
  rules:
  - host: example.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: myservice
            port: 
              number: 80
  tls: # < placing a host in the TLS config will indicate a certificate should be created
  - hosts:
    - example.com
    secretName: myingress-cert # < cert-manager will store the created certificate in this secret.

See: https://cert-manager.io/docs/usage/ingress/

[andormarkus]

Hi @potiuk and @mik-laj

Thank you for the idea to shifting the SSL into the Ingress. We are using AWS Load Balancer Controller which supports ingress side certificates. I think more people might have the same issue who are new into Kubernetes and Airflow. Can we extended Helm chart documentation with mik-laj's recommendation?

I have followed Marc Lamberti's 'Apache Airflow on AWS EKS: The Hands-On Guide' which clearly won't work with the newly released official chart.

Helm chart myconf.yaml looks like this

ingress:
  enabled: true
  web:
    precedingPaths:
      - path: "/*"
        serviceName: "ssl-redirect"
        servicePort: "use-annotation"
    path: "/airflow/*"
    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600

     alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-central-1:XXXXXXXXXX:certificate/XXXXXXXXXXX
     alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
     alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'

[mik-laj]

Can we extended Helm chart documentation with mik-laj's recommendation?

Good idea. Are you willing to submit a PR? I'm happy to help. with review.

I have followed Marc Lamberti's 'Apache Airflow on AWS EKS: The Hands-On Guide' which clearly won't work with the newly released official chart.

CC: @marclamberti

[marclamberti]

Thanks for noticing me :) I will update the course

[potiuk]

Yep. It would be great to submit a pr @andormarkus . And yeah. Ingress is even better than having your own proxy.

[andormarkus]

I'm more than happy to submit a pr, however I'm busy with migration to the official helm chart. Airflow version 2.0.2+ are causing so much headache. See #16013 #16020

[potiuk]

I'm more than happy to submit a pr, however I'm busy with migration to the official helm chart. Airflow version 2.0.2+ are causing so much headache. See #16013 #16020

I see. Did you (by any chance) try 2.1.0 or manually upgrading cncf-kubernetes provider to 1.2.0?

I think there are couple of issues fixed in cncf.kubernetes 1.2.0 version of the provider and at least one looks like might be fixing the labelling problem you explained (https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/index.html#id1) - you can see detailed list of commits for 1.2.0 cncf.kubernetes provider here: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/commits.html#id1 and the one I am thinking about is https://github.com/apache/airflow/commit/be421a6b07c2ae9167150b77dc1185a94812b358

The easiest (if you want to try 1.0.2 with new provider) is to build your own image and extend it by installing the new provider with something like:

FROM apache/airflow:2.0.2-pythonX.Y
RUN pip install --upgrade apache-airflow-providers-cncf-kubernetes

[andormarkus]

Hi @potiuk, I have tried the official helm chart 1.0.0 with Airflow 2.0.1, 2.0.2, 2.1.0 as well. Only 2.0.1 is working, 2.0.2 and 2.1.0 does not work mentioned in the related issues. As I checked Airflow 2.1.0 comes with apache-airflow-providers-cncf-kubernetes==1.2.0

[potiuk]

So we need to take a closer look then! Thanks for verifying.

[kaxil]

Thanks @andormarkus We will take a look at those issues you have created

[Anirudhaagrawal]

My issue is somewhat related. Airflow UI keeps redirecting me to HTTP instead of https.

I was following this. https://airflow.apache.org/docs/apache-airflow/stable/howto/run-behind-proxy.html

I tried adding something like this

 webserver:
  base_url: "https://k8s.foobaa.com/airflow"

But it gave me

airflow:
- webserver: Additional property base_url is not allowed

I am using aws elb with Kong for proxy.

[andormarkus]

@Anirudhaagrawal During my initial comment I was'n fully familiar with the helm chart values file. potiuk and mik-laj recommended: we should not use HTTPS inside kubernetes, it will fail on the liveness and readiness probes.

Recommended traffic flow looks like this: internet --> HTTPS --> K8s ingress controller --> HTTP --> Airflow webserver

my configuration looks like this (AWS EKS with aws-load-balancer-controller):

config:
  webserver:
    base_url: https://k8s.foobaa.com/airflow

ingress:
  enabled: true
  web:
    precedingPaths:
      - path: "/*"
        serviceName: "ssl-redirect"
        servicePort: "use-annotation"
    path: "/airflow/*"

    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600

      alb.ingress.kubernetes.io/certificate-arn: ${aws_acm_certificate_arn}
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
      alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'

[kaxil]

I am going to close this ticket for now with https://github.com/apache/airflow/issues/16010#issuecomment-847929472 as the correct solution. For other issues #16013 #16020 -- We will take a look and comment on those specific tickets

[rdeteix]

Hello

Is it still working ?

I'am on k8s v1.21, helm 1.3.0, airflow 2.2.3. After getting an error about a missing pathType I tried :

  values:
    config:
       webserver:
           base_url: "http://localhost:8080/dev"
    ingress:
      enabled: true
      web:
        precedingPaths:
          - path: "/*"
            pathType: "ImplementationSpecific"
            serviceName: "ssl-redirect"
            servicePort: "use-annotation"
        path: /dev/*
        annotations:
          kubernetes.io/ingress.class: alb
          alb.ingress.kubernetes.io/scheme: internal
          alb.ingress.kubernetes.io/target-type: ip
          alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
          alb.ingress.kubernetes.io/certificate-arn: mycertificate
          alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
          alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'

I let the service to be a cluster ip. My ingress gives me:

Name:             airflow-dev-airflow-ingress
Namespace:        dev
Address:         my-adress
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /*       ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
              /dev/*   airflow-dev-webserver:airflow-ui ()
Annotations:  alb.ingress.kubernetes.io/actions.ssl-redirect:
                {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
              alb.ingress.kubernetes.io/certificate-arn: mycertificate
              alb.ingress.kubernetes.io/listen-ports: [{"HTTP": 80}, {"HTTPS":443}]
              alb.ingress.kubernetes.io/scheme: internal
              alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
              alb.ingress.kubernetes.io/target-type: ip
              kubernetes.io/ingress.class: alb
              meta.helm.sh/release-name: airflow-dev
              meta.helm.sh/release-namespace: dev
Events:
  Type    Reason                  Age   From     Message
  ----    ------                  ----  ----     -------
  Normal  SuccessfullyReconciled  112s  ingress  Successfully reconciled

The ingress manifest that it created is:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: *****
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
    alb.ingress.kubernetes.io/target-type: ip
    kubernetes.io/ingress.class: alb
    meta.helm.sh/release-name: airflow-dev
    meta.helm.sh/release-namespace: dev
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: ssl-redirect
            port:
              name: use-annotation
        path: /*
        pathType: ImplementationSpecific
      - backend:
          service:
            name: airflow-dev-webserver
            port:
              name: airflow-ui
        path: /dev/*
        pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
    - hostname: *****

[andormarkus]

Hi

We are using this config with helm 1.3.0

config:
  webserver:
    base_url: "http://dataengineering.acme.com/airflow"

ingress:
  enabled: true
  web:
    precedingPaths:
      - path: "/*"
        pathType: "ImplementationSpecific"
        serviceName: "ssl-redirect"
        servicePort: "use-annotation"
    path: "/airflow/*"

    annotations:
      external-dns.alpha.kubernetes.io/hostname: ${web_url}
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/group.name: dataengineering
      alb.ingress.kubernetes.io/group.order: '10'
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600

      alb.ingress.kubernetes.io/certificate-arn: ${aws_acm_certificate_arn}
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
      alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'

I think your config.webserver.base_url is incorrect, it needs to be a valid private/public URL

[rdeteix]

Thx for the response. I've already tried it. Which k8s version are you working with ?

I also tried it without the reverse proxy :

    ingress:
      enabled: true
      web:
        precedingPaths:
          - path: "/*"
            pathType: "ImplementationSpecific"
            serviceName: "ssl-redirect"
            servicePort: "use-annotation"
        path: "/*"

        annotations:
          external-dns.alpha.kubernetes.io/hostname: ****
          kubernetes.io/ingress.class: alb
          alb.ingress.kubernetes.io/group.name: airflow
          alb.ingress.kubernetes.io/group.order: '10'
          alb.ingress.kubernetes.io/scheme: internal
          alb.ingress.kubernetes.io/target-type: ip
          alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600

          alb.ingress.kubernetes.io/certificate-arn:  ****
          alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
          alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'

I have:

ts=2022-01-04T18:52:13.592931819Z caller=release.go:397 component=release release=airflow-dev targetNamespace=dev resource=dev:helmrelease/airflow helmVersion=v3 warning="failed to annotate release resources: error: arguments in resource/name form must have a single resource and name\n" phase=annotate

[rdeteix]

The alb controller has changed since v2.3 : https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/tasks/ssl_redirect/ I'll try to make it work/

[dylac]

@andormarkus is the most recent chart you provided still working? I've tried a few versions of it, also updating the ssl-redirect but no luck. Is localhost not acceptable in the base_url? Should it match my external-dns annotation?

At this point I feel I've tried just about every combination of NLB/ALB annotations, enabling the proxy fix vars for gunicorn, and am still struggling with getting Airflow SSL to work on EKS. It's just completely stuck in "redirect hell". Is there no config variation that will simply not perform redirects?

[andormarkus]

Hi @dylac

Do you have running AWS Load Balancer Controller or NGINX Ingress Controller on you EKS cluster?

Your base_url should be the same as your Route53 entry plus path. I have followed Marc Lamberti's course for the initial setup of Airflow. Sorry I got no access to my Airflow configuration anymore because I switched job.

Thanks, Andor

[monircefalo]

Hi @andormarkus , i am using NGINX ingress controller on my eks cluster. Do i need to change anything in your provided solution?