search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.66k stars 4.61k forks source link

If dns-controller encounters an error for a record it won't apply any records #3155

Open blakebarnett opened 6 years ago

blakebarnett commented 6 years ago

kops version: 1.7.0 k8s version: 1.7.1 dns-controller version: 1.6.1

For example: dns-controller-1075714275-pzgnh dns-controller W0807 23:28:26.641579 1 dnscontroller.go:119] Unexpected error in DNS controller, will retry: error querying resource records for zone "zone.it.cannot.access.": error querying resource records for zone "zone.it.cannot.access.": AccessDenied: User: arn:aws:sts::XXXXXXXX:assumed-role/masters.role/i-xxxxxxxxxxx is not authorized to perform: route53:ListResourceRecordSets on resource: arn:aws:route53:::hostedzone/XXXXXXXXXXXX

Once this error is encountered no other DNS changes get applied, which means a bad annotation anywhere in the cluster breaks DNS updates for everyone.

fejta-bot commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta. /lifecycle stale

blakebarnett commented 6 years ago

/lifecycle frozen

blakebarnett commented 6 years ago

I imagine we can close this once we switch to external-dns assuming it doesn't have the same problems.

chrislovecnm commented 6 years ago

@blakebarnett I think fixing dnscomtroller to make this a wanting rather than an error is appropriate