search

aws-controllers-k8s / community

AWS Controllers for Kubernetes (ACK) is a project enabling you to manage AWS services from Kubernetes
https://aws-controllers-k8s.github.io/community/
Apache License 2.0
2.39k stars 253 forks source link

RouteTable update using vpcEndpointID results in CR spec getting converted from vpcEndpointID to GatewayID #1935

Open nnbu opened 10 months ago

nnbu commented 10 months ago

Describe the bug Steps mentioned below describe the issue

Steps to reproduce

  1. Create RouteTable CR without any routes 

    apiVersion: ec2.services.k8s.aws/v1alpha1
kind: RouteTable
metadata:
name: rt1
spec:
vpcID: vpc-03fab91f7a1866946

  2. Add routes to the CR which has vpcEndpointID and apply the CR again

    apiVersion: ec2.services.k8s.aws/v1alpha1
kind: RouteTable
metadata:
labels:
app.kubernetes.io/created-by: cluster-api-provider-aws-ack
app.kubernetes.io/part-of: cluster-api-provider-aws-ack
cluster-name: prad-e2e-test
name: rt1
spec:
routes:
- destinationCIDRBlock: 192.168.16.0/26
 vpcEndpointID: vpce-0d16fc08e2473fc4b
vpcID: vpc-03fab91f7a1866946

  3. After this, CR spec changes and vpcEndpointID gets removed and gatewayID gets added to the CR. Value of gatewayID remains same as vpcEndpointID

apiVersion: ec2.services.k8s.aws/v1alpha1
kind: RouteTable
metadata:
  labels:
    app.kubernetes.io/created-by: cluster-api-provider-aws-ack
    app.kubernetes.io/part-of: cluster-api-provider-aws-ack
    cluster-name: prad-e2e-test
  name: rt1
spec:
  routes:
  - destinationCIDRBlock: 192.168.16.0/26
     gatewayID: vpce-0d16fc08e2473fc4b
  vpcID: vpc-03fab91f7a1866946

Expected outcome CR spec should remain unchanged

Environment

nnbu commented 10 months ago

The real problem here with aws api itself. e.g. I created a new route in existing route-table using following CLI. Notice that I have provided vpc-endpoint-id as argument

$ aws ec2 create-route --route-table-id rtb-0d41902e37988a44b --destination-cidr-block 192.168.16.0/26 --vpc-endpoint-id vpce-0d16fc08e2473fc4b --region us-west-2

But when I describe the route, aws api returns it as gatewayID instead of vpcendpointID as shown below

$ aws ec2 describe-route-tables --region us-west-2 
        {
            "Associations": [],
            "PropagatingVgws": [],
            "RouteTableId": "rtb-0d41902e37988a44b",
            "Routes": [
                {
                    "DestinationCidrBlock": "192.168.16.0/26",
                    "GatewayId": "vpce-0d16fc08e2473fc4b",
                    "Origin": "CreateRoute",
                    "State": "active"
                },
            ...
nnbu commented 10 months ago

When route is added to routeTable CR, sdkUpdate is called and then immediately sdkFind is called. This sdkFind gets GatewayId instead of vpcEndpointId due to the bug in aws cli described above. This is then returned from customUpdateRouteTable and CR spec gets modified from having vpcEndpointId to having GatewayId

Relevant code: https://github.com/aws-controllers-k8s/ec2-controller/blob/main/pkg/resource/route_table/hooks.go#L216

nnbu commented 10 months ago

The same issue does NOT happen for nat-gateway-id, because aws api returns the correct value. e.g.

$ aws ec2 create-route --route-table-id  rtb-06883be7cd0fe6be3 --destination-cidr-block 192.168.16.0/26  --nat-gateway-id nat-0837c191cbd2fa45f --region us-west-2

AWS api to describe route table correctly returns the id as nat-gateway-id (instead of gatewayID)

$ aws ec2 describe-route-tables --region us-west-2
        {
            "Associations": [],
            "PropagatingVgws": [],
            "RouteTableId": "rtb-06883be7cd0fe6be3",
            "Routes": [
                {
                    "DestinationCidrBlock": "192.168.16.0/26",
                    "NatGatewayId": "nat-0837c191cbd2fa45f",
                    "Origin": "CreateRoute",
                    "State": "active"
                },
ack-bot commented 2 months ago

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

ack-bot commented 3 weeks ago

Stale issues rot after 60d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 60d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle rotten