search

kubernetes-sigs / external-dns

Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services
Apache License 2.0
7.69k stars 2.56k forks source link

`failed to sync cache: timed out waiting for the condition` When using `istio-*` source #2314

Closed danthegoodman1 closed 3 years ago

danthegoodman1 commented 3 years ago

What happened: When using external dns v0.9.0, After 20-60 seconds the deployment crashes with failed to sync cache: timed out waiting for the condition.

How to reproduce it (as minimally and precisely as possible):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: external-dns
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: k8s.gcr.io/external-dns/external-dns:v0.9.0
        args:
        - --source=istio-gateway
        - --provider=digitalocean
        - --policy=sync
        env:
        - name: DO_TOKEN
          value: "xxx"

(version 1.11.2) istioctl install

Anything else we need to know?:

Environment:

danthegoodman1 commented 3 years ago

I find these logs in k9s:

│ time="2021-09-21T20:43:31Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:z │
│ alando.org/v1 Sources:[service istio-gateway] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:fals │
│ e Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:digitalocean GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInte │
│ rval:1s GoogleZoneVisibility: DomainFilter:[] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: │
│  AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernet │
│ es/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatConfigFile:/etc/kubernetes/bluecat.json CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ Rcodezer │
│ oTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin Infoblox │
│ WapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml  │
│ InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: TX │
│ TSuffix: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exos │
│ cale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zon │
│ e: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC │
│ 2136BatchChangeSize:50 NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTT │
│ L:0 GoDaddyOTE:false}"                                                                                                                                                                                                     │
│ time="2021-09-21T20:43:31Z" level=info msg="Instantiating new Kubernetes client"                                                                                                                                           │
│ time="2021-09-21T20:43:31Z" level=debug msg="apiServerURL: "                                                                                                                                                               │
│ time="2021-09-21T20:43:31Z" level=debug msg="kubeConfig: "                                                                                                                                                                 │
│ time="2021-09-21T20:43:31Z" level=info msg="Using inCluster-config based on serviceaccount-token"                                                                                                                          │
│ time="2021-09-21T20:43:31Z" level=info msg="Created Kubernetes client https://10.245.0.1:443"                                                                                                                              │
│ time="2021-09-21T20:43:32Z" level=debug msg="service added"                                                                                                                                                                │
│ time="2021-09-21T20:43:32Z" level=debug msg="service added"                                                                                                                                                                │
│ time="2021-09-21T20:43:32Z" level=debug msg="service added"                                                                                                                                                                │
│ time="2021-09-21T20:43:32Z" level=debug msg="service added"                                                                                                                                                                │
│ time="2021-09-21T20:43:32Z" level=debug msg="service added"                                                                                                                                                                │
│ time="2021-09-21T20:43:32Z" level=debug msg="service added"                                                                                                                                                                │
│ time="2021-09-21T20:43:32Z" level=debug msg="service added"                                                                                                                                                                │
│ time="2021-09-21T20:44:33Z" level=fatal msg="failed to sync cache: timed out waiting for the condition"                                                                                                                    │
│ stream closed

However if I use service instead of istio-gateway as the source, I see the following:

│ time="2021-09-21T20:45:37Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:z │
│ time="2021-09-21T20:45:37Z" level=info msg="Instantiating new Kubernetes client"                                                                                                                                           │
│ time="2021-09-21T20:45:37Z" level=debug msg="apiServerURL: "                                                                                                                                                               │
│ time="2021-09-21T20:45:37Z" level=debug msg="kubeConfig: "                                                                                                                                                                 │
│ time="2021-09-21T20:45:37Z" level=info msg="Using inCluster-config based on serviceaccount-token"                                                                                                                          │
│ time="2021-09-21T20:45:37Z" level=info msg="Created Kubernetes client https://10.245.0.1:443"                                                                                                                              │
│ time="2021-09-21T20:45:44Z" level=debug msg="Endpoints generated from DigitalOcean DNS" endpoints="[do-nyc-1.clusterspace.io 0 IN NS  ns1.digitalocean.com;ns2.digitalocean.com;ns3.digitalocean.com [] api.do-nyc-1.clust │
│ time="2021-09-21T20:45:44Z" level=debug msg="No endpoints could be generated from service default/kubernetes"                                                                                                              │
│ time="2021-09-21T20:45:44Z" level=debug msg="No endpoints could be generated from service kube-system/kube-dns"                                                                                                            │
│ time="2021-09-21T20:45:44Z" level=debug msg="No endpoints could be generated from service istio-system/istiod"                                                                                                             │
│ time="2021-09-21T20:45:44Z" level=debug msg="No endpoints could be generated from service istio-system/istio-ingressgateway"                                                                                               │
│ time="2021-09-21T20:45:44Z" level=debug msg="No endpoints could be generated from service cert-manager/cert-manager"                                                                                                       │
│ time="2021-09-21T20:45:44Z" level=debug msg="No endpoints could be generated from service cert-manager/cert-manager-webhook"                                                                                               │
│ time="2021-09-21T20:45:44Z" level=debug msg="Endpoints generated from service: default/api-lb: [api.do-nyc-1.clusterspace.io 0 IN A  143.244.202.37 []]"                                                                   │
│ time="2021-09-21T20:45:44Z" level=debug msg="Skipping endpoint promrelay.com 0 IN A  157.230.201.73 [] because owner id does not match, found: \"\", required: \"default\""
danthegoodman1 commented 3 years ago

Solved, had to add:

- apiGroups: ["", "extensions", "apps", "networking.k8s.io", "networking.istio.io", "authentication.istio.io", "rbac.istio.io", "config.istio.io", "security.istio.io"]
  resources: ["*"]
  verbs: ["*"]

to my ClusterRole