EKS Cluster Subnet - Invalid Preview

stefan01 commented 3 months ago

What happened?

We are creating an EKS Cluster using the pulumi-aws provider with python 3.12. The EKS Cluster already exists and we imported it using pulumi import .... That cluster is associated to 6 subnets and we are changing it to just 3 inside of our pulumi program.

When we run pulumi preview --diff, the output of the subnet_ids inside of vpc_config changes with every run and it often does not match with the desired ids. The output seems like it is random.

Examples:

Run 1:

Run 2:

Run 3:

Property Test

We implemented a property test to check the resulting subnets and we noticed that they are correct. When checking the values inside of a debugger, they are also matching as expected. Only the preview seems to be invalid.

We are assuming that this is an issue inside of the preview rendering of the pulumi cli (?),

Example

cluster.py

# Create EKS Cluster
self.cluster = aws.eks.Cluster(
    resource_name=config.name,
    name=config.name,
    role_arn=role.role.arn,
    vpc_config={
        "subnet_ids": ["subnet-1234567", "subnet-4532456", "subnet-436425145",],   # Example ids, in my case I used real existing subnet ids
        "endpoint_private_access": config.enable_private_access,
        "endpoint_public_access": config.enable_public_access,
    },
    enabled_cluster_log_types=[
        "api",
        "audit",
        "authenticator",
        "controllerManager",
        "scheduler",
    ],
    bootstrap_self_managed_addons=False,
    tags=config.tags,
    opts=child_opts.merge(pulumi.ResourceOptions(
        ignore_changes=["vpcConfig.securityGroupIds"])),
)

property-test.py

def eks_subnet_validator(
    args: StackValidationArgs, report_violation: ReportViolation,
):
    eks: PolicyResource = next(
        filter(
            lambda r: r.resource_type == "aws:eks/cluster:Cluster",
            args.resources,
        ),
    )

    # Check VPC Config
    eks_subnet_ids = list(eks.props["vpcConfig"]["subnetIds"])
    expected_subnet_ids = ["subnet-1234567", "subnet-4532456", "subnet-436425145"]   # Example ids, in my case I used real existing subnet ids

    if set(expected_subnet_ids) != set(eks_subnet_ids):
        report_violation(
            f"""The subnets used for the EKS Cluster are not the correct subnets!
            Excepted: {expected_subnet_ids}, Actual: {eks_subnet_ids}""",
            eks.urn,
        )

Output of `pulumi about`

CLI
Version 3.129.0 Go Version go1.22.6 Go Compiler gc

Plugins KIND NAME VERSION resource aws 6.48.0 language python 3.12 resource random 4.16.3 resource std 1.7.3 resource tls 5.0.4

Host
OS ubuntu Version 22.04 Arch x86_64

Pulumi locates its logs in /tmp by default

Additional context

As we can not publish the real code, we modified it to create a close approximation. The provided code was tested in our condition and resulted in the same issues.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

Frassle commented 3 months ago

I suspect this is a problem with lists vs sets. Probably the aws provider diff method that's not handling this quite right. Moving repos.

t0yv0 commented 3 months ago

Thanks for sending it here. This is a manifestation of a known issue with confusing Set Previews, some combination of:

I will add this item to the workstream and we will follow up here once the fix is available!

cleverguy25 commented 2 months ago

Added to epic https://github.com/pulumi/home/issues/3558

pulumi / pulumi-aws