[Feature Request] Add support for creating CNAMES for .status.loadBalancer.ingress.hostname

[rwaltr]

Hello,

I am attempting to use k8s_gateway for split DNS with an environment that has no consistent external IP, In traefik, I use

--providers.kubernetesingress.ingressendpoint.hostname=traefik.example.com

This allows external-dns to properly populate my public DNS with CNAMEs instead of A records containing private IPs that are not valid on the public web.

However, no .status.loadBalancer.ingress.ip is generated in the ingress option, only .status.loadBalancer.ingress.hostname

Because of this, my ingress options do not populate in k8s_gateway

If k8s_gateway is able to create CNAMEs for the hostnames on an ingress, I would be able to take advantage of that to use the same object in both external-dns and k8s_gateway without issue.

It's possible that there is a solution that allows me to use both as is, however I have not found it quite yet.

Thank you,

[networkop]

AFAIR, k8s_gateway understands status.loadBalancer.address.hostname, see https://github.com/ori-edge/k8s_gateway/blob/f4a6ea896bd3ebf895ce33356685f92b85dcd4ae/kubernetes.go#L495

Can you attach your full ingress resource kubectl get -o yaml , the queries that you're trying to make and what responses to you expect to see?

[rwaltr]

# results of https://github.com/rwaltr/home-ops/blob/cffea6e286c012c760dbab2d516cc6b4d1ebf2e1/k8s/kyz/apps/media-servers/jellyfin/helm-release.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    external-dns.home.arpa/enabled: "true"
    meta.helm.sh/release-name: jellyfin
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2022-10-13T00:27:36Z"
  generation: 5
  labels:
    app.kubernetes.io/instance: jellyfin
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: jellyfin
    helm.sh/chart: app-template-0.2.1
    helm.toolkit.fluxcd.io/name: jellyfin
    helm.toolkit.fluxcd.io/namespace: default
  name: jellyfin
  namespace: default
  resourceVersion: "13535901"
  uid: 69a0ca8f-cdc7-427d-b3b0-4226b91d6cc0
spec:
  ingressClassName: traefik
  rules:
  - host: jellyfin.waltr.tech
    http:
      paths:
      - backend:
          service:
            name: jellyfin
            port:
              number: 8096
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - jellyfin.waltr.tech
status:
  loadBalancer:
    ingress:
    - hostname: kyz.waltr.tech

K8s_gateway deployed at https://github.com/rwaltr/home-ops/blob/cffea6e286c012c760dbab2d516cc6b4d1ebf2e1/k8s/kyz/apps/networking/k8s_gateway/helm-release.yaml

short Dig results

-> dig jellyfin.waltr.tech A @10.22.2.3 +short # Expecting local ingress IP
104.21.22.128 #results in public DNS
172.67.205.10

-> dig jellyfin.waltr.tech CNAME @10.22.2.3 +short # Expecting kyz.waltr.tech
# no results

-> dig jellyfin.default.waltr.tech @10.22.2.3 +short # Expecting IP of jellyfin service
10.22.2.2 # correct response

long dig results

┌─╴ blackphidora@monolith    [☸ admin@kyz: default][🐦kyz] 
└> dig jellyfin.waltr.tech A @10.22.2.3

; <<>> DiG 9.16.33-RH <<>> jellyfin.waltr.tech A @10.22.2.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58545
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: fbffa0d198ae0009 (echoed)
;; QUESTION SECTION:
;jellyfin.waltr.tech.       IN  A

;; ANSWER SECTION:
jellyfin.waltr.tech.    300 IN  A   172.67.205.10
jellyfin.waltr.tech.    300 IN  A   104.21.22.128

;; Query time: 23 msec
;; SERVER: 10.22.2.3#53(10.22.2.3)
;; WHEN: Thu Oct 20 16:52:20 CDT 2022
;; MSG SIZE  rcvd: 130

┌─╴ blackphidora@monolith    [☸ admin@kyz: default][🐦kyz] 
└> dig jellyfin.waltr.tech CNAME @10.22.2.3 # Expecting kyz.waltr.tech

; <<>> DiG 9.16.33-RH <<>> jellyfin.waltr.tech CNAME @10.22.2.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13860
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: b0708301f38af79a (echoed)
;; QUESTION SECTION:
;jellyfin.waltr.tech.       IN  CNAME

;; AUTHORITY SECTION:
waltr.tech.     60  IN  SOA k8s-gateway.networking.waltr.tech. hostmaster.k8s-gateway.networking.waltr.tech. 12345 7200 1800 86400 60

;; Query time: 6 msec
;; SERVER: 10.22.2.3#53(10.22.2.3)
;; WHEN: Thu Oct 20 16:52:47 CDT 2022
;; MSG SIZE  rcvd: 183

┌─╴ blackphidora@monolith    [☸ admin@kyz: default][🐦kyz] 
└> dig jellyfin.default.waltr.tech @10.22.2.3  # Expecting IP of jellyfin service

; <<>> DiG 9.16.33-RH <<>> jellyfin.default.waltr.tech @10.22.2.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32583
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 2f7483a164e57e71 (echoed)
;; QUESTION SECTION:
;jellyfin.default.waltr.tech.   IN  A

;; ANSWER SECTION:
jellyfin.default.waltr.tech. 300 IN A   10.22.2.2

;; Query time: 1 msec
;; SERVER: 10.22.2.3#53(10.22.2.3)
;; WHEN: Thu Oct 20 16:53:08 CDT 2022
;; MSG SIZE  rcvd: 111

[networkop]

ah, I see, so you want k8s_gateway instead of recursively resolving a hostname, to respond you with a CNAME, which your client will then resolve on its own?

[rwaltr]

yes, Either a CNAME, or ideally an A record of traefik's service. Since my DNS starts at my local router, I can put in overrides for my ingress's cname, So when k8s_gateway responds with a Cname, its the same public and internal, but internal is already split DNS. so one change doesn't hurt.

[networkop]

ok, in theory it's possible to make k8s_gateway return a CNAME or resolve a CNAME based on some sort of configuration flag. But I don't understand the ideally an A record of traefik's service part. I understand you're referring to loadBalancer service that traefik puts in front of an ingress controller? Why don't you use providers.kubernetesingress.ingressendpoint.publishedservice and get this automatically published in the ingress status?

[rwaltr]

Thinking back, the ideally an A record of traefik's service was a bad idea to write.

Using providers.kubernetesingress.ingressendpoint.publishedservice does allow k8s_gateway to resolve an A record for an ingress as expected, but at the same time, causes external-dns to start populating public dns with private IPs, even with kubernetesingress.ingressendpoint.hostname enabled on traefik.

[networkop]

ok, so we're down to one option: a global recursive hostname resolution option (on/off). the current behaviour is the same as "on". With "off", k8s_gateway will simply return a CNAME, without any additional A record. Did I get it right?

[rwaltr]

Yes, that sounds correct to me.

[networkop]

ok, got it. I won't have time to implement this in the near future, so marking it as "help wanted" for now.

[rwaltr]

I have found a solution to my problem that nullifies this feature request.

I am able to use providers.kubernetesingress.ingressendpoint.publishedservice set to true, but added an annotation to traefik's service to manually set the DNS records with external-dns.alpha.kubernetes.io/hostname

This was hinted in external-dns's documentation in FAQs, but I missed it until now.

Because of this, I do not need this feature request, But I will leave it in case someone else does need this feature.