search

ori-edge / k8s_gateway

A CoreDNS plugin to resolve all types of external Kubernetes resources
Apache License 2.0
317 stars 66 forks source link

Version 0.4.0 error "Could not sync required resources" but version 0.3.4 works #279

Open rwarford opened 7 months ago

rwarford commented 7 months ago

I installed via Helm chart 2.0.4 (app version 0.3.4) and DNS resolution works correctly. I uninstalled then installed Helm chart 2.1.0 (app version 0.4.0) and DNS resolution no longer works (dig reports SERVFAIL). I've tried Helm chart 2.4.0 and get the same results.

Logs from app version 0.4.0 (note the Could not sync required resources error):

[INFO] plugin/k8s_gateway: Building k8s_gateway controller
[INFO] plugin/k8s_gateway: VirtualServer CRDs are not found. Not syncing VirtualServer resources.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
[INFO] plugin/k8s_gateway: Waiting for controllers to sync
.:1053
[DEBUG] plugin/k8s_gateway: Request 1778848412669144676.1792364358248969471. has not matched any zones [XXXXXXX.com.]
W0407 20:23:30.501935       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:30.502068       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
[INFO] plugin/reload: Running configuration SHA512 = 321cf4ac2f3862a8000e9fb4f5c51a7019dc1d2e0bb1ef25da8ff1beb229fc8c43b57231f2e8bf971eb7d99f54d439ee893119d59937354704a0e61ab22d433f
CoreDNS-1.11.1+k8s_gateway-0.4.0
linux/amd64, go1.21.3, 22e389c
W0407 20:23:30.504228       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:30.504300       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
[DEBUG] plugin/k8s_gateway: Adding index nginx-service.default for service nginx-service
[DEBUG] plugin/k8s_gateway: Adding index k8s-gateway-application.k8s-gateway-system for service k8s-gateway-application
[INFO] 127.0.0.1:59904 - 42794 "HINFO IN 1778848412669144676.1792364358248969471. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.014001293s
W0407 20:23:31.767988       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:31.768041       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
W0407 20:23:31.934310       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:31.934360       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
W0407 20:23:33.888547       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:33.888610       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
W0407 20:23:34.946547       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:34.946679       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
W0407 20:23:38.099568       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:38.099619       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
W0407 20:23:41.340906       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:41.340959       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
W0407 20:23:49.403101       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:49.403211       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
W0407 20:23:50.171164       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:50.171208       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
W0407 20:24:04.470328       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:24:04.470372       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
[DEBUG] plugin/k8s_gateway: Computed Index Keys [nginx-service.default.XXXXXXX.com nginx-service.default]
[INFO] 10.42.232.128:51583 - 6720 "A IN nginx-service.default.XXXXXXX.com. udp 93 false 4096" - - 0 0.000086851s
[ERROR] plugin/errors: 2 nginx-service.default.XXXXXXX.com. A: plugin/k8s_gateway: Could not sync required resources

Logs from app version 0.3.4 (which is working correctly):

[INFO] plugin/k8s_gateway: Building k8s_gateway controller
[INFO] plugin/k8s_gateway: GatewayAPI CRDs are not found. Not syncing GatewayAPI resources.
[INFO] plugin/k8s_gateway: VirtualServer CRDs are not found. Not syncing VirtualServer resources.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
[INFO] plugin/k8s_gateway: Waiting for controllers to sync
.:1053
[INFO] plugin/reload: Running configuration SHA512 = 321cf4ac2f3862a8000e9fb4f5c51a7019dc1d2e0bb1ef25da8ff1beb229fc8c43b57231f2e8bf971eb7d99f54d439ee893119d59937354704a0e61ab22d433f
CoreDNS-1.10.1+k8s_gateway-0.3.4
linux/amd64, go1.20.2, c982ce7
[DEBUG] plugin/k8s_gateway: Request 962279960104462924.4524425992165016576. has not matched any zones [XXXXXXX.com.]
[DEBUG] plugin/k8s_gateway: Adding index nginx-service.default for service nginx-service
[DEBUG] plugin/k8s_gateway: Adding index k8s-gateway-application.k8s-gateway-system for service k8s-gateway-application
[INFO] 127.0.0.1:44227 - 42749 "HINFO IN 962279960104462924.4524425992165016576. udp 56 false 512" NXDOMAIN qr,rd,ra 131 0.012402933s
[INFO] plugin/k8s_gateway: Synced all required resources
[DEBUG] plugin/k8s_gateway: Computed Index Keys [nginx-service.default.XXXXXXX.com nginx-service.default]
[DEBUG] plugin/k8s_gateway: Found 0 matching Ingress objects
[DEBUG] plugin/k8s_gateway: Found 1 matching Service objects
[DEBUG] plugin/k8s_gateway: Computed response addresses [10.100.11.105]
[INFO] 10.100.11.210:32824 - 3653 "A IN nginx-service.default.XXXXXXX.com. udp 93 false 4096" NOERROR qr,aa 138 0.000609899s
koeppj commented 6 months ago

I am experiencing the same issue.

koeppj commented 6 months ago

More on this. Installed the experimental channel CRDs of the Gateway release 0.4.0 and issue resolved.

$kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml

This installs the alpha CRDs for GPCRoute and TLSRoute. I think the plugin fails on the sync call if ALL referenced CRDs are not present (GPCRoute and TLSRoute are excluded in the standard channel).

rwarford commented 6 months ago

More on this. Installed the experimental channel CRDs of the Gateway release 0.4.0 and issue resolved.

$kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml

This installs the alpha CRDs for GPCRoute and TLSRoute. I think the plugin fails on the sync call if ALL referenced CRDs are not present (if is they are excluded in using resource type filter.

I think this must be the case. Seems like it should work with services and ingresses and just ignore routes if the CRDs aren't installed.

Joker9944 commented 2 months ago

Maybe this can help anyone else coming looking for a solution. I only really needed the gateway to watch ingresses.networking.k8s.io. So I just restricted the ClusterRole with a Kustomize patch. This works around the issue at least for now.

https://github.com/Joker9944/k8s-config/blob/5f1ede42e9dbb8f12d9e642c6d7a357ead51a0cd/apps/nameserver-apps/blocky/helm-release.yaml#L215-L278

larivierec commented 1 month ago

Maybe this can help anyone else coming looking for a solution. I only really needed the gateway to watch ingresses.networking.k8s.io. So I just restricted the ClusterRole with a Kustomize patch. This works around the issue at least for now.

https://github.com/Joker9944/k8s-config/blob/5f1ede42e9dbb8f12d9e642c6d7a357ead51a0cd/apps/nameserver-apps/blocky/helm-release.yaml#L215-L278

this is the right solution. the chart should add rbac dynamically, however, it's static: (rbac)[https://github.com/ori-edge/k8s_gateway/blob/master/charts/k8s-gateway/templates/rbac.yaml#L28-L33]

another issue is you can't deactivate it to create your own RBAC with this chart.