NetworkVpcStack: Error occurred: Route53Exception - Rate exceeded

[khris-zeroeyes]

Describe the bug

I encountered the following error in the Network account while deploying TSE-SE v1.10.0-a with LZA v1.10.1:

ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  2/64 | 10:13:13 PM | CREATE_FAILED        | AWS::Route53::HostedZone           | EndpointVpcS3AccesspointEpHostedZone (EndpointVpcS3AccesspointEpHostedZone536BE857) Resource handler returned message: "Error occurred: Route53Exception - Rate exceeded (Service: Route53, Status Code: 400, Request ID: 6af453cb-4f0a-4738-8847-19006f1eb333)" (RequestToken: 01853370-7143-05cf-1235-4d88120e5da9, HandlerErrorCode: null)
    new HostedZone (/codebuild/output/src2204/src/s3/00/source/packages/@aws-accelerator/constructs/lib/aws-route-53/hosted-zone.ts:42:22)
    \_ forEach (/codebuild/output/src2204/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/network-stacks/network-vpc-dns-stack/network-vpc-dns-stack.ts:273:26)
    \_ Array.forEach (<anonymous>)
    \_ NetworkVpcDnsStack.createAdditionalS3Records (/codebuild/output/src2204/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/network-stacks/network-vpc-dns-stack/network-vpc-dns-stack.ts:271:22)
    \_ NetworkVpcDnsStack.createHostedZones (/codebuild/output/src2204/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/network-stacks/network-vpc-dns-stack/network-vpc-dns-stack.ts:169:16)
    \_ NetworkVpcDnsStack.createPrivateHostedZones (/codebuild/output/src2204/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/network-stacks/network-vpc-dns-stack/network-vpc-dns-stack.ts:107:14)
    \_ new NetworkVpcDnsStack (/codebuild/output/src2204/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/network-stacks/network-vpc-dns-stack/network-vpc-dns-stack.ts:47:10)
    \_ createNetworkVpcStacks (/codebuild/output/src2204/src/s3/00/source/packages/@aws-accelerator/accelerator/utils/stack-utils.ts:1043:22)
    \_ createMultiAccountMultiRegionStacks (/codebuild/output/src2204/src/s3/00/source/packages/@aws-accelerator/accelerator/bin/app.ts:191:29)
    \_ processTicksAndRejections (node:internal/process/task_queues:95:5)

Prior to this failure I see these Route53 resource creation requests in progress:

ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcCloudformationEpHostedZone (EndpointVpcCloudformationEpHostedZoneF0BA5B3A) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcS3AccesspointEpHostedZone (EndpointVpcS3AccesspointEpHostedZone536BE857) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEcrDkrEpHostedZone (EndpointVpcEcrDkrEpHostedZone48A98CCA) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcStsEpHostedZone (EndpointVpcStsEpHostedZoneBA517503) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcS3ControlEpHostedZone (EndpointVpcS3ControlEpHostedZone8CA511B1) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcKmsEpHostedZone (EndpointVpcKmsEpHostedZone3C54A278) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEc2messagesEpHostedZone (EndpointVpcEc2messagesEpHostedZone7125A042) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEcrApiEpHostedZone (EndpointVpcEcrApiEpHostedZone6FBEA1EB) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53Resolver::ResolverRule | EndpointOutboundResolverRuleZePiLzaEndpointMadExampleLocal (EndpointOutboundResolverRuleZePiLzaEndpointMadExampleLocal3D864F07) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcSqsEpHostedZone (EndpointVpcSqsEpHostedZone0F1E6669) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcS3EpHostedZone (EndpointVpcS3EpHostedZone469F3968) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcLogsEpHostedZone (EndpointVpcLogsEpHostedZone598F445C) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEksEpHostedZone (EndpointVpcEksEpHostedZone36AF85B9) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcMonitoringEpHostedZone (EndpointVpcMonitoringEpHostedZoneC1BDFD53) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcSsmmessagesEpHostedZone (EndpointVpcSsmmessagesEpHostedZoneB39B7A49) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEc2EpHostedZone (EndpointVpcEc2EpHostedZone7B0297AB) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcSsmEpHostedZone (EndpointVpcSsmEpHostedZone4C564DFD) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcSecretsmanagerEpHostedZone (EndpointVpcSecretsmanagerEpHostedZone4FA05A2F) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:01 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcElasticfilesystemEpHostedZone (EndpointVpcElasticfilesystemEpHostedZone6F7C9D35) 
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcKmsEpHostedZone (EndpointVpcKmsEpHostedZone3C54A278) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcS3ControlEpHostedZone (EndpointVpcS3ControlEpHostedZone8CA511B1) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEc2EpHostedZone (EndpointVpcEc2EpHostedZone7B0297AB) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcSqsEpHostedZone (EndpointVpcSqsEpHostedZone0F1E6669) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcCloudformationEpHostedZone (EndpointVpcCloudformationEpHostedZoneF0BA5B3A) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcSecretsmanagerEpHostedZone (EndpointVpcSecretsmanagerEpHostedZone4FA05A2F) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcSsmmessagesEpHostedZone (EndpointVpcSsmmessagesEpHostedZoneB39B7A49) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcMonitoringEpHostedZone (EndpointVpcMonitoringEpHostedZoneC1BDFD53) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEcrApiEpHostedZone (EndpointVpcEcrApiEpHostedZone6FBEA1EB) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcS3AccesspointEpHostedZone (EndpointVpcS3AccesspointEpHostedZone536BE857) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcLogsEpHostedZone (EndpointVpcLogsEpHostedZone598F445C) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEksEpHostedZone (EndpointVpcEksEpHostedZone36AF85B9) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcElasticfilesystemEpHostedZone (EndpointVpcElasticfilesystemEpHostedZone6F7C9D35) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53Resolver::ResolverRule | EndpointOutboundResolverRuleZePiLzaEndpointMadExampleLocal (EndpointOutboundResolverRuleZePiLzaEndpointMadExampleLocal3D864F07) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  0/64 | 10:13:02 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcSsmEpHostedZone (EndpointVpcSsmEpHostedZone4C564DFD) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  2/64 | 10:13:03 PM | CREATE_IN_PROGRESS   | AWS::Route53Resolver::ResolverRule | EndpointOutboundResolverRuleZePiLzaEndpointMadExampleLocal (EndpointOutboundResolverRuleZePiLzaEndpointMadExampleLocal3D864F07) Eventual consistency check initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  2/64 | 10:13:05 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEcrDkrEpHostedZone (EndpointVpcEcrDkrEpHostedZone48A98CCA) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  2/64 | 10:13:07 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcS3EpHostedZone (EndpointVpcS3EpHostedZone469F3968) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  2/64 | 10:13:07 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcStsEpHostedZone (EndpointVpcStsEpHostedZoneBA517503) Resource creation Initiated
ze-pi-lza-NetworkVpcDnsStack-273354641138-us-east-1 |  2/64 | 10:13:07 PM | CREATE_IN_PROGRESS   | AWS::Route53::HostedZone           | EndpointVpcEc2messagesEpHostedZone (EndpointVpcEc2messagesEpHostedZone7125A042) Resource creation Initiated

To Reproduce Use a minimally customized TSE-SE reference architecture configuration, with more than the default share of VPC endpoints exposed.

Expected behavior A clear and concise description of what you expected to happen.

Please complete the following information about the solution:

• [x] Version: v1.10.1

To get the version of the solution, you can look at the description of the created AWS CloudFormation stack used to install the LZA (AWSAccelerator-InstallerStack). For example, "(SO0199) Landing Zone Accelerator on AWS. Version 1.5.1.". If the description does not contain the version information, you can look at the Parameters of the stack for the RepositoryBranchName as that should contain the version number.

• [x] Region: us-east-1
• [x] Was the solution modified from the version published on this repository? yes, it's a slight variant of the TSE-SE reference architecture
• [x] If the answer to the previous question was yes, are the changes available on GitHub? no
• [x] Have you checked your service quotas for the services this solution uses? yes
• [x] Were there any errors in the CloudWatch Logs? yes, I've attached the relevant snippet

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context Add any other context about the problem here.

[khris-zeroeyes]

I was able to work around this issue by adding the VPC endpoints in batches