pulumi / pulumi-aws

An Amazon Web Services (AWS) Pulumi resource package, providing multi-language access to AWS
Apache License 2.0
466 stars 157 forks source link

EC2 instance always shown as needing an `update` after `pulumi up` #3449

Open Kazy opened 9 months ago

Kazy commented 9 months ago

What happened?

We've imported our current infrastructure into Pulumi. When running pulumi up, all our EC2 instances are shown as requiring an update, but when going into the diff, nothing changes. Accepting the update and running pulumi up again does the same thing, meaning we have no way of resolving the drift. Doing pulumi refresh shows no change.

This doesn't happen when using version 6.8 of this provider, the bug starts happening from 6.9 up to the latest 6.22.

Example

First run:

λ pulumi up
Previewing update (prod):
     Type                          Name                                            Plan       Info
     pulumi:pulumi:Stack           shuttle-prod                                               1 message
     └─ shuttle:Shuttle            shuttle-prod
        ├─ shuttle:Admin           shuttle-prod-admin
        │  └─ shuttle:Node         shuttle-prod-admin-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-admin-node-ec2-instance            update
        ├─ shuttle:Controller      shuttle-prod-ctl
        │  └─ shuttle:Node         shuttle-prod-ctl-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-ctl-node-ec2-instance              update
        ├─ shuttle:DatabaseVm      shuttle-prod-shared-database
        │  └─ shuttle:Node         shuttle-prod-shared-database-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-shared-database-node-ec2-instance  update
        ├─ shuttle:Builder         shuttle-prod-builder
        │  └─ shuttle:Node         shuttle-prod-builder-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-builder-node-ec2-instance          update
        ├─ shuttle:Master          shuttle-prod-master
        │  └─ shuttle:Node         shuttle-prod-master-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-master-node-ec2-instance           update
        └─ shuttle:Provisionner    shuttle-prod-provisioner
           └─ shuttle:Node         shuttle-prod-provisioner-node
 ~            └─ aws:ec2:Instance  shuttle-prod-provisioner-node-ec2-instance      update

Diagnostics:
  pulumi:pulumi:Stack (shuttle-prod):
    Running using profile: shuttle-prod

Resources:
    ~ 6 to update
    117 unchanged

Do you want to perform this update? yes
Updating (prod):
     Type                          Name                                            Status           Info
     pulumi:pulumi:Stack           shuttle-prod                                                     1 message
     └─ shuttle:Shuttle            shuttle-prod
        ├─ shuttle:Controller      shuttle-prod-ctl
        │  └─ shuttle:Node         shuttle-prod-ctl-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-ctl-node-ec2-instance              updated (1s)
        ├─ shuttle:Admin           shuttle-prod-admin
        │  └─ shuttle:Node         shuttle-prod-admin-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-admin-node-ec2-instance            updated (1s)
        ├─ shuttle:Builder         shuttle-prod-builder
        │  └─ shuttle:Node         shuttle-prod-builder-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-builder-node-ec2-instance          updated (1s)
        ├─ shuttle:Master          shuttle-prod-master
        │  └─ shuttle:Node         shuttle-prod-master-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-master-node-ec2-instance           updated (1s)
        ├─ shuttle:DatabaseVm      shuttle-prod-shared-database
        │  └─ shuttle:Node         shuttle-prod-shared-database-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-shared-database-node-ec2-instance  updated (1s)
        └─ shuttle:Provisionner    shuttle-prod-provisioner
           └─ shuttle:Node         shuttle-prod-provisioner-node
 ~            └─ aws:ec2:Instance  shuttle-prod-provisioner-node-ec2-instance      updated (1s)

Diagnostics:
  pulumi:pulumi:Stack (shuttle-prod):
    Running using profile: shuttle-prod

Outputs:
    auroraConnInfo: "env/PROD_CONTROL_DB_CONNECTION_INFO"

Resources:
    ~ 6 updated
    117 unchanged

Duration: 14s

Second run:

Λ pulumi up
Previewing update (prod):
     Type                          Name                                            Plan       Info
     pulumi:pulumi:Stack           shuttle-prod                                               1 message
     └─ shuttle:Shuttle            shuttle-prod
        ├─ shuttle:DatabaseVm      shuttle-prod-shared-database
        │  └─ shuttle:Node         shuttle-prod-shared-database-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-shared-database-node-ec2-instance  update
        ├─ shuttle:Master          shuttle-prod-master
        │  └─ shuttle:Node         shuttle-prod-master-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-master-node-ec2-instance           update
        ├─ shuttle:Builder         shuttle-prod-builder
        │  └─ shuttle:Node         shuttle-prod-builder-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-builder-node-ec2-instance          update
        ├─ shuttle:Controller      shuttle-prod-ctl
        │  └─ shuttle:Node         shuttle-prod-ctl-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-ctl-node-ec2-instance              update
        ├─ shuttle:Admin           shuttle-prod-admin
        │  └─ shuttle:Node         shuttle-prod-admin-node
 ~      │     └─ aws:ec2:Instance  shuttle-prod-admin-node-ec2-instance            update
        └─ shuttle:Provisionner    shuttle-prod-provisioner
           └─ shuttle:Node         shuttle-prod-provisioner-node
 ~            └─ aws:ec2:Instance  shuttle-prod-provisioner-node-ec2-instance      update

Diagnostics:
  pulumi:pulumi:Stack (shuttle-prod):
    Running using profile: shuttle-prod

Resources:
    ~ 6 to update
    117 unchanged

Do you want to perform this update? no
confirmation declined, not proceeding with the update

Output of pulumi about

10:15:17 λ pulumi about
CLI
Version      3.101.1
Go Version   go1.21.5
Go Compiler  gc

Plugins
NAME       VERSION
aws        6.22.2
aws        6.22.2
aws        6.9.0
aws        6.8.0
aws        6.8.0
awsx       2.5.0
awsx       2.5.0
cloudinit  1.4.0
cloudinit  1.4.0
command    0.9.2
command    0.9.2
docker     4.5.1
docker     4.5.1
docker     3.6.1
docker     3.6.1
nodejs     unknown
std        1.6.2
std        1.6.2
tls        5.0.1
tls        5.0.1

Host
OS       arch
Version
Arch     x86_64

This project is written in nodejs: executable='/run/user/1000/fnm_multishells/4365_1708412546054/bin/node' version='v21.2.0'

Current Stack: organization/shuttle/prod

TYPE                                                 URN
<SKIPPED>

Found no pending operations associated with prod

Backend
Name           nocsik-fixe
URL            s3://<SKIPPED>
User           noscik
Organizations
Token type     personal

Dependencies:
NAME               VERSION
@pulumi/command    0.9.2
remeda             1.43.0
@types/node        20.11.19
yaml               2.3.4
@pulumi/aws        6.22.2
@pulumi/awsx       2.5.0
zod                3.22.4
@pulumi/pulumi     3.106.0
@pulumi/std        1.6.2
@pulumi/tls        5.0.1
@pulumi/cloudinit  1.4.0

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

iwahbe commented 9 months ago

Hi @Kazy. Thanks for filing a bug. I don't see this behavior when creating an aws.ec2.Instance and running pulumi up. Can you post steps so I can reproduce this on my computer.

Kazy commented 9 months ago

Hi @Kazy. Thanks for filing a bug. I don't see this behavior when creating an aws.ec2.Instance and running pulumi up. Can you post steps so I can reproduce this on my computer.

Unfortunately I don't think I can. This only happened on one of our env that I imported but not the other. The instance was originally created using Terraform and then imported, if that's help. Considering I can't provide a repro, I'll gladly help test anything that might help. We can also hop on a call if needed, let me know.

jackwtech commented 8 months ago

I have a similar problem when creating ec2 instance using launch template that has user_data. The diff doesn't show any changes.

Below code can be used to re-produce the problem. (Workaround is adding ignore_changes=["user_data"] to avoid the update)

import base64
import pulumi
import pulumi_aws as aws

ubuntu = aws.ec2.get_ami(
    most_recent=True,
    filters=[
        aws.ec2.GetAmiFilterArgs(
            name="name",
            values=["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"],
        ),
        aws.ec2.GetAmiFilterArgs(
            name="virtualization-type",
            values=["hvm"],
        ),
    ],
    owners=["099720109477"],
)

default = aws.ec2.DefaultVpc(
    "default",
    tags={
        "Name": "Default VPC",
    },
)

user_data = """#!/bin/bash
echo "Hello World"
"""

encoded_user_data = base64.b64encode(user_data.encode()).decode()

launch_template = aws.ec2.LaunchTemplate(
    "default_launch_template",
    name_prefix="default",
    image_id=ubuntu.id,
    instance_type="t3.micro",
    user_data=encoded_user_data,
)

web = aws.ec2.Instance(
    "web",
    launch_template=aws.ec2.InstanceLaunchTemplateArgs(
        id=launch_template.id,
        version="$Latest",
    ),
    tags={
        "Name": "HelloWorld",
    },
    # opts=pulumi.ResourceOptions(ignore_changes=["user_data"]),
)
t0yv0 commented 2 months ago

Still reproduces for me. I was looking at the upstream definition for ec2.Instance and it's a little interesting.

            "user_data": {
                Type:          schema.TypeString,
                Optional:      true,
                Computed:      true,
                ConflictsWith: []string{"user_data_base64"},
                DiffSuppressFunc: func(k, old, new string, d *schema.ResourceData) bool {
                    // Sometimes the EC2 API responds with the equivalent, empty SHA1 sum
                    // echo -n "" | shasum
                    if (old == "da39a3ee5e6b4b0d3255bfef95601890afd80709" && new == "") ||
                        (old == "" && new == "da39a3ee5e6b4b0d3255bfef95601890afd80709") {
                        return true
                    }
                    return false
                },
                StateFunc: func(v interface{}) string {
                    switch v := v.(type) {
                    case string:
                        return userDataHashSum(v)
                    default:
                        return ""
                    }
                },
                ValidateFunc: validation.StringLenBetween(0, 16384),
            },

In the statefile I see

                    "userData": "09c5cea10b191a7b79a797e9a0673dc0c89a29aa",

Curious if da39a3ee5e6b4b0d3255bfef95601890afd80709 magic is now 09c5cea10b191a7b79a797e9a0673dc0c89a29aa magic that similarly needs ignoring, but could not find any reference for this. If this theory is right the issue should reproduce upstream in plain TF also.

pulumi about:

CLI          
Version      3.130.0
Go Version   go1.22.6
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  aws     6.52.0
language  python  unknown

Host     
OS       darwin
Version  14.6.1
Arch     arm64

This project is written in python: executable='/Users/anton/Library/Mobile Documents/com~apple~CloudDocs/pulumi-bugs/pulumi-aws-3449/venv/bin/python' version='3.10.13'

Current Stack: anton-pulumi-corp/pulumi-aws-3449/dev

TYPE                                   URN
pulumi:pulumi:Stack                    urn:pulumi:dev::pulumi-aws-3449::pulumi:pulumi:Stack::pulumi-aws-3449-dev
pulumi:providers:aws                   urn:pulumi:dev::pulumi-aws-3449::pulumi:providers:aws::default_6_52_0
aws:ec2/launchTemplate:LaunchTemplate  urn:pulumi:dev::pulumi-aws-3449::aws:ec2/launchTemplate:LaunchTemplate::default_launch_template
aws:ec2/defaultVpc:DefaultVpc          urn:pulumi:dev::pulumi-aws-3449::aws:ec2/defaultVpc:DefaultVpc::default
aws:ec2/instance:Instance              urn:pulumi:dev::pulumi-aws-3449::aws:ec2/instance:Instance::web

Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/anton-pulumi-corp
User           anton-pulumi-corp
Organizations  anton-pulumi-corp, moolumi, demo, pulumi
Token type     personal

Dependencies:
NAME        VERSION
pip         24.2
pulumi_aws  6.52.0
setuptools  75.1.0
wheel       0.44.0

Pulumi locates its logs in /var/folders/gd/3ncjb1lj5ljgk8xl5ssn_gvc0000gn/T/com.apple.shortcuts.mac-helper// by default
t0yv0 commented 2 months ago

I have filed https://github.com/hashicorp/terraform-provider-aws/issues/39476 as it indeed reproduces upstream and best be fixed there. You can help by up-voting this issue.

flostadler commented 2 months ago

@t0yv0 09c5cea10b191a7b79a797e9a0673dc0c89a29aa is the sha1 of:

#!/bin/bash
echo "Hello World"