search

kubernetes-sigs / cluster-api-provider-aws

Kubernetes Cluster API Provider AWS provides consistent deployment and day 2 operations of "self-managed" and EKS Kubernetes clusters on AWS.
http://cluster-api-aws.sigs.k8s.io/
Apache License 2.0
626 stars 542 forks source link

:bug: wait for lb dns name to propagate before resolving #5033

Open r4f4 opened 1 week ago

r4f4 commented 1 week ago

What type of PR is this?

/kind bug

What this PR does / why we need it:

Instead of trying to resolve the primary LB DNS name right after its creation, wait for it to propagate so the resolution is most likely to succeed.

This fixes an issue where the first "no such host" cached dns response with high TTL would make CAPA spin for minutes (as high as 15!) waiting for the DNS name to resolve even though it had already propagated a few minutes after the first attempt.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #5032

Special notes for your reviewer:

I couldn't find a more elegant way to solve this other than a sleep after the LB is created. I wanted to add a retryAfterDuration here right after the DNS name is set and before the name resolution is attempted but it would involve somehow saving state of the timestamp in between reconcile loops.

Checklist:

Release note:

Fixed a possible issue with long wait times for primary Load Balancer DNS name resolution.
k8s-ci-robot commented 1 week ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign nrb for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
k8s-ci-robot commented 1 week ago

Hi @r4f4. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
r4f4 commented 1 week ago

Openshift e2e tests show no regressions for cases with low TTL.

For example in this run the DNS name is resolved right after the wait is done:

time="2024-06-22T13:57:30Z" level=debug msg="I0622 13:57:30.384416     354 loadbalancer.go:70] \"Reconciling load balancers\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\""
[...]
time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.237999     354 loadbalancer.go:128] \"Created new network load balancer for apiserver\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.238015     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
[...]
time="2024-06-22T13:57:46Z" level=debug msg="I0622 13:57:46.238804     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:58:01Z" level=debug msg="I0622 13:58:01.238765     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:58:16Z" level=debug msg="I0622 13:58:16.238794     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:58:31Z" level=debug msg="I0622 13:58:31.238326     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:58:46Z" level=debug msg="I0622 13:58:46.238639     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:59:01Z" level=debug msg="I0622 13:59:01.238991     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:59:16Z" level=debug msg="I0622 13:59:16.238665     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:59:31Z" level=debug msg="I0622 13:59:31.238731     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:59:31Z" level=debug msg="I0622 13:59:31.327379     354 loadbalancer.go:1723] \"creating target group\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" group=<"
[...]
time="2024-06-22T13:59:32Z" level=debug msg="I0622 13:59:32.391053     354 awscluster_controller.go:291] \"Looking up IP address for DNS\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" dns=\"ci-op-kjphm6tx-2d061-qplhx-int-dd788fa810aeb43e.elb.us-east-1.amazonaws.com\""
time="2024-06-22T13:59:32Z" level=debug msg="I0622 13:59:32.617668     354 s3.go:256] \"Created bucket\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" bucket_name=\"openshift-bootstrap-data-ci-op-kjphm6tx-2d061-qplhx\""

and we can see that the secondary LB is reconciling while we wait for the dns name:

time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.237959     354 loadbalancer.go:448] \"Created network load balancer\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" dns-name=\"ci-op-kjphm6tx-2d061-qplhx-int-dd788fa810aeb43e.elb.us-east-1.amazonaws.com\""
time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.237988     354 loadbalancer.go:451] \"applying load balancer DNS to result\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" dns=\"ci-op-kjphm6tx-2d061-qplhx-int-dd788fa810aeb43e.elb.us-east-1.amazonaws.com\""
time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.237999     354 loadbalancer.go:128] \"Created new network load balancer for apiserver\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.238015     354 loadbalancer.go:134] \"Waiting for network load balancer dns name to propagate\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-int\""
time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.784769     354 loadbalancer.go:448] \"Created network load balancer\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" dns-name=\"ci-op-kjphm6tx-2d061-qplhx-ext-e309fac33c3c5cb9.elb.us-east-1.amazonaws.com\""
time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.784798     354 loadbalancer.go:451] \"applying load balancer DNS to result\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" dns=\"ci-op-kjphm6tx-2d061-qplhx-ext-e309fac33c3c5cb9.elb.us-east-1.amazonaws.com\""
time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.784809     354 loadbalancer.go:128] \"Created new network load balancer for apiserver\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" api-server-lb-name=\"ci-op-kjphm6tx-2d061-qplhx-ext\""
time="2024-06-22T13:57:31Z" level=debug msg="I0622 13:57:31.857784     354 loadbalancer.go:1723] \"creating target group\" controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-kjphm6tx-2d061-qplhx\" reconcileID=\"d8375aca-40d9-44e0-bebf-281438fc4afe\" cluster=\"openshift-cluster-api-guests/ci-op-kjphm6tx-2d061-qplhx\" group=<"
richardcase commented 2 days ago

/ok-to-test

r4f4 commented 1 day ago

/test pull-cluster-api-provider-aws-e2e-blocking