Closed rwaltr closed 10 months ago
AFAIR, k8s_gateway understands status.loadBalancer.address.hostname
, see https://github.com/ori-edge/k8s_gateway/blob/f4a6ea896bd3ebf895ce33356685f92b85dcd4ae/kubernetes.go#L495
Can you attach your full ingress resource kubectl get -o yaml
, the queries that you're trying to make and what responses to you expect to see?
# results of https://github.com/rwaltr/home-ops/blob/cffea6e286c012c760dbab2d516cc6b4d1ebf2e1/k8s/kyz/apps/media-servers/jellyfin/helm-release.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
external-dns.home.arpa/enabled: "true"
meta.helm.sh/release-name: jellyfin
meta.helm.sh/release-namespace: default
creationTimestamp: "2022-10-13T00:27:36Z"
generation: 5
labels:
app.kubernetes.io/instance: jellyfin
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: jellyfin
helm.sh/chart: app-template-0.2.1
helm.toolkit.fluxcd.io/name: jellyfin
helm.toolkit.fluxcd.io/namespace: default
name: jellyfin
namespace: default
resourceVersion: "13535901"
uid: 69a0ca8f-cdc7-427d-b3b0-4226b91d6cc0
spec:
ingressClassName: traefik
rules:
- host: jellyfin.waltr.tech
http:
paths:
- backend:
service:
name: jellyfin
port:
number: 8096
path: /
pathType: Prefix
tls:
- hosts:
- jellyfin.waltr.tech
status:
loadBalancer:
ingress:
- hostname: kyz.waltr.tech
K8s_gateway deployed at https://github.com/rwaltr/home-ops/blob/cffea6e286c012c760dbab2d516cc6b4d1ebf2e1/k8s/kyz/apps/networking/k8s_gateway/helm-release.yaml
short Dig results
-> dig jellyfin.waltr.tech A @10.22.2.3 +short # Expecting local ingress IP
104.21.22.128 #results in public DNS
172.67.205.10
-> dig jellyfin.waltr.tech CNAME @10.22.2.3 +short # Expecting kyz.waltr.tech
# no results
-> dig jellyfin.default.waltr.tech @10.22.2.3 +short # Expecting IP of jellyfin service
10.22.2.2 # correct response
long dig results
┌─╴ blackphidora@monolith [☸ admin@kyz: default][🐦kyz]
└> dig jellyfin.waltr.tech A @10.22.2.3
; <<>> DiG 9.16.33-RH <<>> jellyfin.waltr.tech A @10.22.2.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58545
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: fbffa0d198ae0009 (echoed)
;; QUESTION SECTION:
;jellyfin.waltr.tech. IN A
;; ANSWER SECTION:
jellyfin.waltr.tech. 300 IN A 172.67.205.10
jellyfin.waltr.tech. 300 IN A 104.21.22.128
;; Query time: 23 msec
;; SERVER: 10.22.2.3#53(10.22.2.3)
;; WHEN: Thu Oct 20 16:52:20 CDT 2022
;; MSG SIZE rcvd: 130
┌─╴ blackphidora@monolith [☸ admin@kyz: default][🐦kyz]
└> dig jellyfin.waltr.tech CNAME @10.22.2.3 # Expecting kyz.waltr.tech
; <<>> DiG 9.16.33-RH <<>> jellyfin.waltr.tech CNAME @10.22.2.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13860
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: b0708301f38af79a (echoed)
;; QUESTION SECTION:
;jellyfin.waltr.tech. IN CNAME
;; AUTHORITY SECTION:
waltr.tech. 60 IN SOA k8s-gateway.networking.waltr.tech. hostmaster.k8s-gateway.networking.waltr.tech. 12345 7200 1800 86400 60
;; Query time: 6 msec
;; SERVER: 10.22.2.3#53(10.22.2.3)
;; WHEN: Thu Oct 20 16:52:47 CDT 2022
;; MSG SIZE rcvd: 183
┌─╴ blackphidora@monolith [☸ admin@kyz: default][🐦kyz]
└> dig jellyfin.default.waltr.tech @10.22.2.3 # Expecting IP of jellyfin service
; <<>> DiG 9.16.33-RH <<>> jellyfin.default.waltr.tech @10.22.2.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32583
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 2f7483a164e57e71 (echoed)
;; QUESTION SECTION:
;jellyfin.default.waltr.tech. IN A
;; ANSWER SECTION:
jellyfin.default.waltr.tech. 300 IN A 10.22.2.2
;; Query time: 1 msec
;; SERVER: 10.22.2.3#53(10.22.2.3)
;; WHEN: Thu Oct 20 16:53:08 CDT 2022
;; MSG SIZE rcvd: 111
ah, I see, so you want k8s_gateway instead of recursively resolving a hostname, to respond you with a CNAME, which your client will then resolve on its own?
yes, Either a CNAME, or ideally an A record of traefik's service. Since my DNS starts at my local router, I can put in overrides for my ingress's cname, So when k8s_gateway responds with a Cname, its the same public and internal, but internal is already split DNS. so one change doesn't hurt.
ok, in theory it's possible to make k8s_gateway return a CNAME or resolve a CNAME based on some sort of configuration flag.
But I don't understand the ideally an A record of traefik's service
part. I understand you're referring to loadBalancer
service that traefik puts in front of an ingress controller? Why don't you use providers.kubernetesingress.ingressendpoint.publishedservice
and get this automatically published in the ingress status?
Thinking back, the ideally an A record of traefik's service
was a bad idea to write.
Using providers.kubernetesingress.ingressendpoint.publishedservice
does allow k8s_gateway to resolve an A record for an ingress as expected, but at the same time, causes external-dns to start populating public dns with private IPs, even with kubernetesingress.ingressendpoint.hostname
enabled on traefik.
ok, so we're down to one option: a global recursive hostname resolution option (on/off). the current behaviour is the same as "on". With "off", k8s_gateway will simply return a CNAME, without any additional A record. Did I get it right?
Yes, that sounds correct to me.
ok, got it. I won't have time to implement this in the near future, so marking it as "help wanted" for now.
I have found a solution to my problem that nullifies this feature request.
I am able to use providers.kubernetesingress.ingressendpoint.publishedservice
set to true, but added an annotation to traefik's service to manually set the DNS records with external-dns.alpha.kubernetes.io/hostname
This was hinted in external-dns's documentation in FAQs, but I missed it until now.
Because of this, I do not need this feature request, But I will leave it in case someone else does need this feature.
Hello,
I am attempting to use k8s_gateway for split DNS with an environment that has no consistent external IP, In traefik, I use
This allows
external-dns
to properly populate my public DNS withCNAME
s instead ofA
records containing private IPs that are not valid on the public web.However, no
.status.loadBalancer.ingress.ip
is generated in the ingress option, only.status.loadBalancer.ingress.hostname
Because of this, my ingress options do not populate in
k8s_gateway
If k8s_gateway is able to create
CNAME
s for the hostnames on an ingress, I would be able to take advantage of that to use the same object in bothexternal-dns
andk8s_gateway
without issue.It's possible that there is a solution that allows me to use both as is, however I have not found it quite yet.
Thank you,