Ingress hosts not being resolved

[xtitoris]

I'm testing out this plugin, but having a lot of trouble getting it to resolve Ingress hosts. I have installed k8s_gateway via install-clusterwide.yml, adjusting k8s_gateway value, and I have metallb with traefik ingress controller configured. Having these resources set up for my test app:

kind: Service
apiVersion: v1
metadata:
  name: test-http-service
  namespace: test-ns
spec:
  selector:
    app: test-app
  ports:
    - port: 80
      targetPort: 80
      name: port-http
---
kind: Service
apiVersion: v1
metadata:
  name: test-service
  namespace: test-ns
spec:
  selector:
    app: test-app
  type: LoadBalancer
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
      name: port-tcp
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test-ns
spec:
  rules:
  - host: test.mydomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: test-http-service
            port:
              number: 80

Both the load balanced service and ingress are working correctly, and I can access them. If I do nslookup test-service.test-ns.mydomain.com 192.168.20.23 (192.168.20.23 being the address of external-dns service), then I receive a correct IP for the test-service. However, if I then try nslookup test.mydomain.com 192.168.20.23, then I get NXDOMAIN instead.

There are no errors in logs either

kubectl logs excoredns-66877775dd-dgfs9 -n kube-system
[INFO] plugin/k8s_gateway: Building k8s_gateway controller
[INFO] plugin/k8s_gateway: GatewayAPI CRDs are not found. Not syncing GatewayAPI resources.
[INFO] plugin/k8s_gateway: VirtualServer CRDs are not found. Not syncing VirtualServer resources.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
[INFO] plugin/k8s_gateway: Waiting for controllers to sync
.:53
[INFO] plugin/reload: Running configuration SHA512 = 6a5d0de0a0958bc6590ca3e6617550be19560b69cf5d8b40e174020327e81b8189184aad4c0dca40432b7285ee7099106920bf03c2c8f6beeb9d6e81c08d6152
CoreDNS-1.9.3+k8s_gateway-0.3.1
linux/amd64, go1.19, 3c866bb
[INFO] 127.0.0.1:51430 - 51762 "HINFO IN 1172258836029527161.9105035297959244524. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.029187973s
[INFO] plugin/k8s_gateway: Synced all required resources
[INFO] 192.168.20.6:11703 - 13181 "A IN test.mydomain.com. udp 29 false 512" NXDOMAIN qr,rd 134 0.000218949s
[INFO] 192.168.20.6:8645 - 9746 "A IN test.mydomain.com. udp 33 false 512" NXDOMAIN qr,rd 138 0.000244904s

Any help with diagnosing the issue will be appreciated.

[networkop]

what do you see when you do kubectl get ing test-ingress -n test-ns ? Can you also add debug plugin to the k8s_gateway's configmap to increase verbosity?

[xtitoris]

$ kubectl get ing -n test-ns
NAME             CLASS    HOSTS               ADDRESS   PORTS   AGE
test-ingress     <none>   test.mydomain.com             80      2d

$ kubectl logs excoredns-66877775dd-hkgjs -n kube-system
[INFO] plugin/k8s_gateway: Building k8s_gateway controller
[INFO] plugin/k8s_gateway: GatewayAPI CRDs are not found. Not syncing GatewayAPI resources.
[INFO] plugin/k8s_gateway: VirtualServer CRDs are not found. Not syncing VirtualServer resources.
.:53
[INFO] plugin/reload: Running configuration SHA512 = 247898e842e1fd303dd62b85d77bbae3cc06e279a86b5f69258a0bd4dbcf615a3a5c86db4dcb7b608280110b10e5aac32058aca4fa02087ff1d9b7e9c0405fdd
CoreDNS-1.9.3+k8s_gateway-0.3.1
linux/amd64, go1.19, 3c866bb
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
[INFO] plugin/k8s_gateway: Waiting for controllers to sync
[DEBUG] plugin/k8s_gateway: Request 1940618674593095557.622131207284608147. has not matched any zones [mydomain.com.]
[INFO] 127.0.0.1:57163 - 20280 "HINFO IN 1940618674593095557.622131207284608147. udp 56 false 512" NXDOMAIN qr,rd,ra 131 0.084827128s
[DEBUG] plugin/k8s_gateway: Adding index test.mydomain.com for ingress test-ingress
[DEBUG] plugin/k8s_gateway: Adding index external-dns.kube-system for service external-dns
[DEBUG] plugin/k8s_gateway: Adding index test-service.test-ns for service test-service
[DEBUG] plugin/k8s_gateway: Adding index traefik.traefik for service traefik
[INFO] plugin/k8s_gateway: Synced all required resources
[DEBUG] plugin/k8s_gateway: Computed Index Keys [test.mydomain.com test]
[DEBUG] plugin/k8s_gateway: Found 1 matching Ingress objects
[DEBUG] plugin/k8s_gateway: Found 0 matching Service objects
[DEBUG] plugin/k8s_gateway: Computed response addresses []
[INFO] 192.168.20.6:30974 - 31757 "A IN test.mydomain.com. udp 29 false 512" NXDOMAIN qr,rd 134 0.000234858s

Looks like it can't find the address for the ingress Here's the output for traefik service

$ kubectl get service -n traefik
NAME      TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
traefik   LoadBalancer   10.152.183.147   192.168.20.20   80:31446/TCP,443:32063/TCP   341d

Accessing this external-ip with a HOST header correctly returns the corresponding services response.

[xtitoris]

Thank you for the push in the right direction. The issue was with traefik configuration, and rather poor documentation. In the latest version traefik had a lot of breaking configuration changes. kubernetes.ingressEndpoint.useDefaultPublishedService option was changed to providers.kubernetesIngress.publishedService.enabled So the updated helm chart portion should look like this:

providers:
  kubernetesIngress:
    publishedService:
      enabled: true