Could not sync required resources

[onedr0p]

Hi :wave:

I am running this and getting an error. My service account has the right access, unless I am missing something. Using the kubeconfig I mention below I can list ingresses and services just fine with kubectl.

I can verify if I use a kubeconfig with full access it works fine.

Error

[INFO] 192.168.1.245:27521 - 60295 "A IN sonarr.devbu.io. udp 33 false 512" NOERROR - 0 0.000103497s
[ERROR] plugin/errors: 2 sonarr.devbu.io. A: plugin/k8s_gateway: Could not sync required resources
[INFO] 192.168.1.245:12301 - 6304 "A IN sonarr.devbu.io. udp 33 false 512" NOERROR - 0 0.000079323s
[ERROR] plugin/errors: 2 sonarr.devbu.io. A: plugin/k8s_gateway: Could not sync required resources
[INFO] 192.168.1.245:64464 - 36655 "A IN sonarr.devbu.io. udp 33 false 512" NOERROR - 0 0.000066486s
[ERROR] plugin/errors: 2 sonarr.devbu.io. A: plugin/k8s_gateway: Could not sync required resources
[INFO] 192.168.42.14:36831 - 25931 "AAAA IN github.com. udp 39 true 2048" NOERROR qr,rd,ra 125 0.014133757s
[INFO] 192.168.42.14:65282 - 47432 "A IN github.com. udp 39 true 2048" NOERROR qr,rd,ra 54 0.014829848s
[INFO] 192.168.42.14:36831 - 64151 "A IN github-releases.githubusercontent.com. udp 66 true 2048" NOERROR qr,rd,ra 267 0.014893559s
[INFO] 192.168.42.14:65282 - 950 "AAAA IN github-releases.githubusercontent.com. udp 66 true 2048" NOERROR qr,rd,ra 161 0.014985431s
E0715 11:24:41.546622   39870 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: Get "https://192.168.42.5:6443/apis/networking.k8s.io/v1beta1/ingresses?limit=500&resourceVersion=0": dial tcp 192.168.42.5:6443: connect: host is down
[INFO] 192.168.1.245:37991 - 12560 "A IN sonarr.devbu.io. udp 33 false 512" NOERROR - 0 0.000085847s
[ERROR] plugin/errors: 2 sonarr.devbu.io. A: plugin/k8s_gateway: Could not sync required resources
[INFO] 192.168.1.245:64391 - 19269 "A IN sonarr.devbu.io. udp 33 false 512" NOERROR - 0 0.000082698s
[ERROR] plugin/errors: 2 sonarr.devbu.io. A: plugin/k8s_gateway: Could not sync required resources
I0715 11:24:42.847208   39870 trace.go:205] Trace[436340495]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167 (15-Jul-2021 11:24:12.844) (total time: 30002ms):
Trace[436340495]: [30.002660732s] [30.002660732s] END
E0715 11:24:42.847288   39870 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.42.5:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.42.5:6443: i/o timeout

RBAC

apiVersion: v1
kind: ServiceAccount
metadata:
  name: home-dns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: home-dns
rules:
- apiGroups:
  - ""
  resources:
  - services
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - extensions
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: home-dns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: home-dns
subjects:
- kind: ServiceAccount
  name: home-dns
  namespace: kube-system

kubeconfig

---
apiVersion: v1
kind: Config
clusters:
- name: home
  cluster:
    certificate-authority-data: <redacted>
    server: https://192.168.42.5:6443
contexts:
- name: home
  context:
    cluster: home
    user: home-dns
users:
- name: home-dns
  user:
    token: <redacted>
current-context: home

[networkop]

are you installing this as a helm chart?

[onedr0p]

No, I am using this as a coredns plugin on my router.

[networkop]

it must be the RBAC issue then. your role + roleBinding look fine. Check that coredns itself is running with the home-dns serviceAccount and in kube-system namespace.

[onedr0p]

This instance of coredns w/ k8s_gateway is not running in my cluster. It is running on my router. The purpose of coredns on my router is to get the ingresses from my k8s cluster and serve them as dns records to my network.

[networkop]

oh yeah, sry, I've missed the fact that you're running outside of the cluster. can you share your corefile as well?

[onedr0p]

.:53 {
  bind 192.168.1.1 127.0.0.1 ::1
  errors
  log
  reload
  loadbalance
  prometheus :9153
  cache
  k8s_gateway domain.tld {
    resources Ingress
    ttl 30
    kubeconfig /usr/local/etc/coredns/kubeconfig
    fallthrough
  }
  forward . tls://1.1.1.1 tls://1.0.0.1 {
    tls_servername cloudflare-dns.com
  }
}

[onedr0p]

🤦🏼 PEBKAC issue, my kube api ip was hella wrong (off by one number) in the kubeconfig