Confusing diffs when unknowns are passed into set or list-nested blocks

t0yv0 commented 5 months ago

What happened?

There is machinery in MakeTerraformInputs specifically makeTerraformUnknown that tries to substitute reasonably shaped empty values for unknowns when the schema indicates lists or sets. Unfortunately this is involved in Check turnaround, so that Pulumi providers start swallowing these unknowns even at Check level, and definitely continue swallowing them during diffs.

Primarily this leads to surprising and unreliable Diff behavior where unknowns are replaced by empties.

But occasionally it also leads to substantial problems; in case of pulumi/pulumi-aws#3835 this uncovered a latent bug making the provider panic on this empty input that would not have otherwise occur in a normal program.

Example

Customer repro:

package main

import (
    "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
    "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/iam"
    "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/rds"
    "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/secretsmanager"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {

        assumeRolePolicy, _ := iam.GetPolicyDocument(ctx, &iam.GetPolicyDocumentArgs{
            Statements: []iam.GetPolicyDocumentStatement{
                {
                    Actions:    []string{"sts:AssumeRole"},
                    Principals: []iam.GetPolicyDocumentStatementPrincipal{{Type: "Service", Identifiers: []string{"lambda.amazonaws.com"}}},
                },
            },
        })
        role, _ := iam.NewRole(ctx, "myrole", &iam.RoleArgs{
            AssumeRolePolicy: pulumi.String(assumeRolePolicy.Json),
        })

        vpc, _ := ec2.NewVpc(ctx, "Vpc", &ec2.VpcArgs{
            CidrBlock: pulumi.String("10.0.0.0/16"),
        })
        subnet, _ := ec2.NewSubnet(ctx, "subnet", &ec2.SubnetArgs{
            VpcId:            vpc.ID(),
            CidrBlock:        pulumi.String("10.0.2.0/24"),
            AvailabilityZone: pulumi.String("us-east-2a"),
        })
        anotherSubnet, _ := ec2.NewSubnet(ctx, "another-subnet", &ec2.SubnetArgs{
            VpcId:            vpc.ID(),
            CidrBlock:        pulumi.String("10.0.1.0/24"),
            AvailabilityZone: pulumi.String("us-east-1a"),
        })

        secret, err := secretsmanager.NewSecret(ctx, "secret", &secretsmanager.SecretArgs{})
        if err != nil {
            panic(err)
        }

        if 1+2 == 3 {
            rds.NewProxy(ctx, "proxy", &rds.ProxyArgs{
                VpcSubnetIds: pulumi.StringArray{
                    subnet.ID(),
                    anotherSubnet.ID(),
                },
                Auths: secret.Arn.ApplyT(func(arn string) []rds.ProxyAuth {
                    return []rds.ProxyAuth{
                        {
                            AuthScheme: pulumi.StringRef("SECRETS"),
                            IamAuth:    pulumi.StringRef("DISABLED"),
                            SecretArn:  &arn,
                        },
                    }
                }).(rds.ProxyAuthArrayInput),
                RoleArn:      role.Arn,
                EngineFamily: pulumi.String("MYSQL"),
            })
        }
        return nil
    })
}

Auths receives a secret.Arn which is unknown. However Check translates it to a known value.

Output of `pulumi about`

N/A

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

VenelinMartinov commented 5 months ago

Thanks for the report and the repro here!

VenelinMartinov commented 3 months ago

@iwahbe I've done some investigation on this around https://github.com/pulumi/pulumi-terraform-bridge/issues/2032 which is likely a duplicated/very related.

I'll take over if that's ok

VenelinMartinov commented 3 months ago

https://github.com/pulumi/pulumi-terraform-bridge/pull/2061 fixes this

pulumi-bot commented 2 months ago

This issue has been addressed in PR #2061 and shipped in release v3.88.0.

pulumi / pulumi-terraform-bridge