Cannot successfully register targets if VPC subnets are different from VPC CIDR block associations

[nullren]

Describe the bug the controller cannot register targets in targetgroupbinding with aws api error

ValidationError: The Availability Zone is required for IP address '10.4.40.87' because it is not in the VPC

this is because the registerPodEndpoints function is looking up cidr blocks associated to the vpc and not the actual subnets which is what the aws api compares.

Steps to reproduce

create vpc and associate more cidr blocks than subnets

$ aws ec2 describe-vpcs --vpc-ids vpc-000000000000 | jq -r '.Vpcs[] | .CidrBlockAssociationSet[] | .CidrBlock'
10.1.0.0/16
10.2.0.0/16
10.3.0.0/16
10.4.0.0/16

then add a few subnets to the vpc

$ aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-000000000000" | jq -r '.Subnets[] | .CidrBlock'
10.1.0.0/19
10.2.0.0/19

try to manually add a target to the target group

$ aws elbv2 register-targets --target-group-arn arn:aws:elasticloadbalancing:us-east-1:000000000000:targetgroup/k8s-default-targetgroupname-0000000/0000000000000 --targets Id=10.4.46.99,Port=80

An error occurred (ValidationError) when calling the RegisterTargets operation: The Availability Zone is required for IP address '10.4.46.99' because it is not in the VPC

Expected outcome this is fixed when adding AvailabilityZone=all to the request and what should happen in the initial check

Environment

• AWS Load Balancer controller version: 2.7.2
• Kubernetes version
• Using EKS (yes/no), if so version? yes

Additional Context:

[nullren]

i've started a pr to draft out some ideas here but to also on my end to unblock my work.

one thing i'm ensure of is the "source of truth" for what the problem is, it seems there are a few things going on here:

• does the register-targets api actually look to just the subnets for discovery?
• is the difference between what the describe-vpcs subnets and the subnets i can see from describe-subnets something that should be reconciled via something like sharing subnets between accounts?
• how important is autodiscovery here and are we going to cause problems by setting AvailabilityZone=all for these? is there a reason we should not always set it?