Closed spwx closed 7 months ago
Thanks for submitting - will investigate.
Taking a look, I'm wondering if the issue here is that the VPC is in a different account/region than the Arkime Cluster. AWS error messages can be cryptic and I could see the error message being thrown in that scenario. It seems unlikely that the problem is with the particular AZ.
Tested cross-region and cross-account adds; we do handle them more gracefully and a different error message. See: https://github.com/arkime/aws-aio/blob/main/manage_arkime/commands/vpc_add.py#L83
Taking a closer look at the log file, it appears that the VPC Endpoint Service created for the Arkime Cluster was incompatible with all three subnets in the user's VPC, not just a single one of them, and attempted to create a VPC Endpoint in each of them multiple times.
OK - so one guess is something wacky with permissions to add Gateway LB VPC Endpoints to the GWLB Service [1]. This seems pretty unlikely as the error message isn't permission related.
My other guess is that we must ensure that our Gateway Load Balancer has targets to receive traffic (Capture Nodes) in every AZ that's sending traffic to it [2]. In this scenario, the user has traffic sources in AZs that we don't have Capture Nodes, which is somehow known by the GWLB and it prevents creating VPC Endpoints in those AZs. However... we have cross-zone balancing turned on which is supposed to resolve issues like this [3]... I'll try and test this scenario tomorrow, as it seems more likely than the first.
[1] https://github.com/arkime/aws-aio/blob/main/cdk-lib/capture-stacks/capture-nodes-stack.ts#L222 [2] https://docs.aws.amazon.com/elasticloadbalancing/latest/gateway/target-groups.html#registered-targets [3] https://github.com/arkime/aws-aio/blob/main/cdk-lib/capture-stacks/capture-nodes-stack.ts#L51
User confirmed via Slack that they have subnets in 6 AZs, and their logs seem to indicate that three of them are failing. By default, our Capture VPC only has subnets in 2 AZs [1]. It could be that CloudFormation is only trying 3 AZs and they just happen to be three that we don't have Capture Subnets in, and if it tried all of 6 of them at once 4 would fail. Alternatively, something more interesting could be happening here.
An easy way to test is to up the number of AZs and hosts in our Demo Stacks to replicate this scenario. Will pursue.
[1] https://github.com/arkime/aws-aio/blob/main/manage_arkime/core/capacity_planning.py#L19
Just thinking aloud, but we may need to move to a larger fleet of smaller Capture Nodes to have presence in each AZ. This appears to be exactly what Cross-Zone Balancing [1] is designed for, where as long as the hosts in the entire target group have enough aggregate capacity to handle the load the GWLB will spread traffic from a "hot" AZ across them in a reasonable way and prevent that zone's Capture Node from going under.
The tradeoff is increased latency and a cross-zone dependency, both of which seem like a reasonable sacrifice in this scenario.
[1] https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-cross-zone.html
Just need to make sure the cross-zone balancing LBing is sticky, having packets from the same session end up in different AZs would be not good ™️
Just need to make sure the cross-zone balancing LBing is sticky, having packets from the same session end up in different AZs would be not good ™️
Thanks for the note. We already have cross-zone balancing enabled for our Capture VPC, so if things currently work then we should be good to go? Would be a good idea to confirm that that do work.
Looks like stickiness is easy to set [1], but we're not currently setting it [2] (oops).
[1] https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_TargetGroupAttribute.html [2] https://github.com/arkime/aws-aio/blob/main/cdk-lib/capture-stacks/capture-nodes-stack.ts#L49
oops, I swear I checked this before, but maybe that was somewhere else or an old version. :(
It looks like there's a few options for ensuring that the Capture VPC is present in all AZs [1] [2]. We'll need to query this before coming up with our capacity plan.
Also - should be obvious but this new way of provisioning the Capture VPC will result in Cfn deployment failures if run against existing stacks, so we'll need to bump the semver for the package.
[1] https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-public-parameters-global-infrastructure.html [2] https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2/client/describe_availability_zones.html
oops, I swear I checked this before, but maybe that was somewhere else or an old version. :(
Actually, it turns out 5-tuple stickiness is the default for GWLB's, it's just that the setting is configured on the Target Group rather than the LB itself and the docs were a bit confusing on that front. So it has been working all along, AFAIK.
OK - think I got this one figured out, and have posted a fix - https://github.com/arkime/aws-aio/pull/152
Code merged. Reaching out to original user to confirm resolution of issue.
Still waiting for user to retry. Closing this issue and will open a new one if further problems are encountered.
Good day,
Have a fresh install of this great project. Cluster creation completed without error. However, unable to add a capture VPC, due to the following error:
Resource handler returned message: "The VPC endpoint service com.amazonaws.vpce.us-east-1.vpce-svc-01f0fa7502b9b4481 does not support the availability zone of the subnet: subnet-08a4d799ec3c08a58.
subnet-08a4d799ec3c08a58
is in theus-east-1f
availability zone.CLI output below, and error log output attached.
Thanks!
Manage Arkime Log
manage_arkime_failure.log
Manage Arkime CLI Output