search

pulumi / pulumi-aws

An Amazon Web Services (AWS) Pulumi resource package, providing multi-language access to AWS
Apache License 2.0
464 stars 155 forks source link

Ghost updates with 5.42.0 #2647

Closed devnev closed 1 year ago

devnev commented 1 year ago

What happened?

After updating to 5.42.0, we experienced ghost updates, where the resource was marked as to be updated in preview and up, but there were no changes to the properties in the diff, and the update remained even after application. A revert of only @pulumi/aws to 5.41.0 (in package.json and yarn.lock), and no other changes, fixed the issue, with the preview showing only the provider version downgrade and no other changes - and no further changes once that was applied.

Expected Behavior

No update for unmodified resources, applying update makes update go away

Steps to reproduce

The allowElbChangesPolicy in the following code was one of the resources with a ghost update

// Policy that allows ECS to assume a role
const allowEcsPolicy: aws.iam.PolicyDocument = {
  Version: '2012-10-17',
  Statement: [
    {
      Effect: 'Allow',
      Action: 'sts:AssumeRole',
      Principal: {
        Service: 'ecs.amazonaws.com',
      },
    },
  ],
};

// ECS Services declared on other repos will consume this
export const ecsServiceRole = new aws.iam.Role(
  `${stack}-role-ecs`,
  {
    description: 'Managed by Pulumi - do not edit manually',
    assumeRolePolicy: allowEcsPolicy,
  },
  {provider}
);

// Allow ECS Service role to change ELB eg target groups
// See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ecs.html
const allowElbChanges: aws.iam.PolicyDocument = {
  Version: '2012-10-17',
  Statement: [
    {
      Effect: 'Allow',
      Action: [
        'elasticloadbalancing:DeregisterInstancesFromLoadBalancer',
        'elasticloadbalancing:DeregisterTargets',
        'elasticloadbalancing:Describe*',
        'elasticloadbalancing:RegisterInstancesWithLoadBalancer',
        'elasticloadbalancing:RegisterTargets',
        'ec2:Describe*',
        'ec2:AuthorizeSecurityGroupIngress',
      ],
      Resource: '*',
    },
  ],
};

const allowElbChangesPolicy = new aws.iam.Policy(
  `${stack}-role-ecs-elb-policy`,
  {
    name: `${stack}-role-ecs-elb-policy`,
    description: 'Managey by pulumi - do not edit manually',
    policy: allowElbChanges,
  },
  {provider}
);

new aws.iam.RolePolicyAttachment(
  `${stack}-role-ecs-policy-attach`,
  {
    role: ecsServiceRole,
    policyArn: allowElbChangesPolicy.arn,
  },
  {provider}
);

Output of pulumi about

I'm not going to paste the resource URNs, but here's the rest

CLI
Version      3.72.2
Go Version   go1.20.5
Go Compiler  gc

Plugins
NAME          VERSION
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
nodejs        unknown
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0

Host
OS       darwin
Version  12.6.5
Arch     x86_64

Found no pending operations associated with develop-infra

Backend
Name           Marks-MacBook-Pro.local
URL            s3://...
User           marknevill
Organizations

Pulumi locates its logs in /var/folders/sz/rcyd6rg51cbb4qzqsvzyn92m0000gn/T/ by default
warning: Failed to get information about the project runtime: unknown method About for service pulumirpc.LanguageRuntime
warning: Failed to get information about the Pulumi program's dependencies: unknown method GetProgramDependencies for service pulumirpc.LanguageRuntime

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

guineveresaenger commented 1 year ago

Hi @devnev - thank you for filing this issue.

If possible, can you help us out a bit more?

When you say

and the update remained even after application. do you mean this resource continued to be marked as an update, even after performing the ghost update?

Do you experience this behavior on the most recent version of Pulumi - could you try updating your Pulumi CLI and see if this persists?

Finally, it would really help us out if you can send us a minimum viable repro that we can run on our end to find a root cause.

danielrbradley commented 1 year ago

Hi @devnev just to add to what Guin asked - there's no provider or stack variables defined - are you using any specific configuration in the explicit provider variable?

Please could you also share what is shown in the console for you during the preview and the up - including the detailed diff.

Thanks!

devnev commented 1 year ago

Hi, thanks for the ping. We've now tried updating pulumi everywhere - notably renovate bot isn't able to update the github acction's version string automatically - and the issue has gone away.

For more background, below is the diff from the output.

~ aws:iam/policy:Policy: (update)
        [id=arn:aws:iam::123456789012:policy/develop-infra-role-ecs-elb-policy]
        [urn=urn:pulumi:develop-infra::product-infra::aws:iam/policy:Policy::develop-infra-role-ecs-elb-policy]
        [provider=urn:pulumi:develop-infra::product-infra::pulumi:providers:aws::aws::75edc29f-b49a-4325-b2f0-0d37171a79d1]
        description: "Managey by pulumi - do not edit manually"
        name       : "develop-infra-role-ecs-elb-policy"
        path       : "/"
        policy     : (json) {
            Statement: [
                [0]: {
                    Action  : [
                        [0]: "elasticloadbalancing:DeregisterInstancesFromLoadBalancer"
                        [1]: "elasticloadbalancing:DeregisterTargets"
                        [2]: "elasticloadbalancing:Describe*"
                        [3]: "elasticloadbalancing:RegisterInstancesWithLoadBalancer"
                        [4]: "elasticloadbalancing:RegisterTargets"
                        [5]: "ec2:Describe*"
                        [6]: "ec2:AuthorizeSecurityGroupIngress"
                    ]
                    Effect  : "Allow"
                    Resource: "*"
                }
            ]
            Version  : "2012-10-17"
        }

    ~ aws:iam/user:User: (update)
        [id=develop-infra-ci-deploy-user]
        [urn=urn:pulumi:develop-infra::product-infra::aws:iam/user:User::develop-infra-ci-deploy-user]
        [provider=urn:pulumi:develop-infra::product-infra::pulumi:providers:aws::aws::75edc29f-b49a-4325-b2f0-0d371a79d1]
        forceDestroy: false
        name        : "develop-infra-ci-deploy-user"
        path        : "/"
    ~ aws:iam/policy:Policy: (update)
        [id=arn:aws:iam::123456789012:policy/develop-infra-ci-state-update-policy-b7e9421]
        [urn=urn:pulumi:develop-infra::product-infra::aws:iam/policy:Policy::develop-infra-ci-state-update-policy]
        [provider=urn:pulumi:develop-infra::product-infra::pulumi:providers:aws::aws::75edc29f-b49a-4325-b2f0-0d37171a79d1]
        name      : "develop-infra-ci-state-update-policy-b7e9421"
        path      : "/"
        policy    : (json) {
            Statement: [
                [0]: {
                    Action  : [
                        [0]: "s3:PutObject"
                        [1]: "s3:GetObject"
                        [2]: "s3:DeleteObject"
                    ]
                    Effect  : "Allow"
                    Resource: [
                        [0]: "arn:aws:s3:::***/.pulumi/*"
                    ]
                }
            ]
            Version  : "2012-10-17"
        }

The provider is configured as

export const provider = new aws.Provider('aws', {
  region: defaultRegion,
  defaultTags: {
    tags,
  },
});

In CI there were additional errors that we did not see when we successfully reproduced the "ghost update" issue locally:

error: 1 error occurred:
    * updating urn:pulumi:develop-infra::product-infra::aws:iam/policy:Policy::develop-infra-role-ecs-elb-policy: 1 error occurred:
    * reading IAM Policy (arn:aws:iam::123456789012:policy/develop-infra-role-ecs-elb-policy): AccessDenied: User: arn:aws:iam::123456789012:user/develop-infra-ci-deploy-user is not authorized to perform: iam:GetPolicy on resource: policy arn:aws:iam::123456789012:policy/develop-infra-role-ecs-elb-policy because no identity-based policy allows the iam:GetPolicy action
    status code: 403, request id: 3df44795-2eea-409f-8396-4ba1b29e5ce7

error: 1 error occurred:
    * updating urn:pulumi:develop-infra::product-infra::aws:iam/user:User::develop-infra-ci-deploy-user: 1 error occurred:
    * reading IAM User (develop-infra-ci-deploy-user): AccessDenied: User: arn:aws:iam::123456789012:user/develop-infra-ci-deploy-user is not authorized to perform: iam:GetUser on resource: user develop-infra-ci-deploy-user because no identity-based policy allows the iam:GetUser action
    status code: 403, request id: 4b88c730-2990-40d5-9f2f-0a868efd409a

error: 1 error occurred:
    * updating urn:pulumi:develop-infra::product-infra::aws:iam/policy:Policy::develop-infra-ci-state-update-policy: 1 error occurred:
    * reading IAM Policy (arn:aws:iam::123456789012:policy/develop-infra-ci-state-update-policy-b7e9421): AccessDenied: User: arn:aws:iam::123456789012:user/develop-infra-ci-deploy-user is not authorized to perform: iam:GetPolicy on resource: policy arn:aws:iam::123456789012:policy/develop-infra-ci-state-update-policy-b7e9421 because no identity-based policy allows the iam:GetPolicy action
    status code: 403, request id: 6120762d-fd25-422e-ac83-35286590371a
devnev commented 1 year ago

Spoke to soon - we're seeing the issue again, even after pulumi upgrade. I'm going to downgrade @pulumi/aws again and see if that fixes it again.

danielrbradley commented 1 year ago

I've not managed to reproduce this locally myself yet. A few more follow-on questions and thoughts:

  1. Does the diff persist on subsequent updates after the initial update after the upgrade?
  2. Does the diff still occur for a repro program tha doesn't use defaultTags on the provider config?
  3. It should be possible to see the raw output from the provider's diff by using verbose logging.
devnev commented 1 year ago

We seem to have moved past this being an issue, and I can't reproduce it any more. As best as I can tell, upgrades changed the handling of default tags, that caused permission errors, and those caused updates without diffs... or the deafult tags caused updates without diffs, not quite sure on that. I'm pretty sure I also had the same diffs running locally where I have sufficient permissions, but not sure why that would have been the case. Confounding this is the various pulumi versions that were flying around.

Would you prefer we dig further, or OK with closing the issue?

devnev commented 1 year ago

Coming back to this, we're still encountering related issues, although not ghost updates. We've now downgraded back to @pulumi/aws v5.41.0. Mainly, the upgrade seemed to cause the provider's default tags to be propagated onto all sorts of resources all of a sudden, which triggered a bunch of authorization errors. Most of that could have been resolved, except for updating tags for KMS, where even a user with a policy allowing all actions on all resources gets "not authorized to perform: kms:TagResource on resource".

mikhailshilkov commented 1 year ago

I think it all checks out - 5.42.0 introduced a fix for default tags in https://github.com/pulumi/pulumi-aws/pull/2585 which it sounds like is causing downstream issues for you.

Although, I'm not exactly sure what our actions on this issue can be. Should we close it as you suggested earlier @devnev? Will you be able to work past he upgrade (and then to 6.x)?

devnev commented 1 year ago

I think we've managed to work past it, yeah. Closing, thanks