search

kubernetes-sigs / external-dns

Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services
Apache License 2.0
7.72k stars 2.57k forks source link

failed to sync *v1.Node (but I didn't request --source=node) #3169

Open iameli opened 2 years ago

iameli commented 2 years ago

What happened:

I ran external-dns like so:

export EXTERNAL_DNS_KUBECONFIG="./kubeconfig"

external-dns \
        --once \
        --log-level=debug \
        --log-format=text \
        --policy=upsert-only \
        --server="https://redacted:443" \
        --provider=cloudflare \
        --registry=txt \
        --interval=5m0s \
        --txt-owner-id=chi \
        --annotation-filter=livepeer.live/external-dns=true \
        --source=service \
        --source=ingress \
        --dry-run

And then it times out trying to get nodes:

INFO[0000] config: {APIServerURL:https://redacted:443 KubeConfig:./kubeconfig RequestTimeout:30s DefaultTargets:[] ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter:livepeer.live/external-dns=true LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:cloudflare GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatDNSConfiguration: BluecatConfigFile:/etc/kubernetes/bluecat.json BluecatDNSView: BluecatGatewayHost: BluecatRootZone: BluecatDNSServerName: BluecatDNSDeployType:no-deploy BluecatSkipTLSVerify:false CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: InfobloxCreatePTR:false InfobloxCacheDuration:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:upsert-only Registry:txt TXTOwnerID:chi TXTPrefix: TXTSuffix: Interval:5m0s MinEventSyncInterval:5s Once:true DryRun:true UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json}
INFO[0000] running in dry-run mode. No changes to DNS records will be made.
INFO[0000] Instantiating new Kubernetes client
DEBU[0000] apiServerURL: https://redacted:443
DEBU[0000] kubeConfig: ./kubeconfig
INFO[0000] Using kubeConfig
INFO[0000] Created Kubernetes client https://redacted:443
FATA[0060] failed to sync *v1.Node: context deadline exceeded

What you expected to happen:

  1. I don't have permission to list nodes in this (multi-tenant) cluster. I didn't request --source=node or --publish-node-ip or anything like that. Why is it requesting nodes? 
    ▶ kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "token-AA8moJwCcZU2TiKXwsuS" cannot list resource "nodes" in API group "" at the cluster scope
  2. When I do request nodes, I get a 403 error back from kubectl. Why is external-dns timing out?

Versions:

> external-dns --version
v0.13.1
> kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-01-19T17:23:01Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
alebedev87 commented 2 years ago

This is not a bug. You specified service as the source type, by default all the service types are considered as valid sources, including NodePort services. The target for the DNS record generated for the service of type NodePort will contain the IPs of all the nodes, hence the need to request Node resource from the api server. Excluding NodePort services from the service types is supposed to help to not try to request Node resource.

ChipWolf commented 1 year ago

This is not a bug. You specified service as the source type, by default all the service types are considered as valid sources, including NodePort services. The target for the DNS record generated for the service of type NodePort will contain the IPs of all the nodes, hence the need to request Node resource from the api server. Excluding NodePort services from the service types is supposed to help to not try to request Node resource.

Same error when using --service-type-filter=LoadBalancer --source=service on my end.

iameli commented 1 year ago

Yeah, from glancing through the code the request for nodes appears to be unconditional; unless I'm misunderstanding something --source=service will always require a search for nodes.

Seems like better behavior would be to not require nodes but then print an error if there are any NodePort or headless ClusterIP services that would require node information?

FWIW, for my use case, we were able to just migrate entirely to using --source=ingress which is now working fine.

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

iameli commented 1 year ago

/remove-lifecycle rotten

Bug isn't a huge deal, but my comment here would still be an improvement in behavior: https://github.com/kubernetes-sigs/external-dns/issues/3169#issuecomment-1327855220

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 9 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

iameli commented 9 months ago

/remove-lifecycle stale

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

iameli commented 6 months ago

/remove-lifecycle stale

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

mentos1386 commented 3 months ago

/remove-lifecycle stale

I have encountered this as well. Maybe updating the docs that the service source is not supported when namespaced would help.

WowSuchRicky commented 2 weeks ago

I also encountered an offshoot of this issue in a setup of my own, #4834. Agree that the docs could be better here.