Managed to get into a broken state with update in progress which couldn't cancel

[alexbowers]

I ended up in a state where I was unable to run any updates, and running cancel would also error for me.

Expected behavior

Either the update can run, or the cancel should work.

Context (Environment)

example update preview: https://app.pulumi.com/shopblocks/shared/main/previews/aa44f44d-9a97-46b5-a536-b4ac7125e9ed

Error:

error: [409] another update is currently in progress or was interrupted. Try again later.) Try again later
    for more information about this error, see https://pulumi.io/oa5g1o
To learn more about possible reasons and resolution, visit https://www.pulumi.com/docs/troubleshooting/#conflict```

When running pulumi cancel:

$ pulumi cancel
This will irreversibly cancel the currently running update for 'main'!
Please confirm that this is what you'd like to do by typing `main`: main
error: [409] Conflict: The Update has already completed

This happened multiple times going back and forth between the update and cancel commands, so it isn't that the lease was released between the update and cancel happening.

I also couldn't see anything that looked out of place on the stack export.

I eventually ran a refresh which had no changes to make, let that run through (even though it did nothing), and then update worked after that point.

[meagancojocar]

Hi @alexbowers thank you so much for reporting this! We are going to try and repro it, if it happens again let us know so we can investigate in a live state.

[blampe]

@alexbowers can you tell us the output of pulumi about?

[alexbowers]

Version      3.76.1
Go Version   go1.20.6
Go Compiler  gc

Plugins
NAME    VERSION
aws     5.42.0
awsx    1.0.2
docker  3.6.1
nodejs  unknown

Host
OS       darwin
Version  13.4.1
Arch     arm64

This project is written in nodejs: executable='/Users/alexbowers/.nvm/versions/node/v19.6.1/bin/node' version='v19.6.1'

Current Stack: shopblocks/shared/main

TYPE                                                    URN
pulumi:pulumi:Stack                                     urn:pulumi:main::shared::pulumi:pulumi:Stack::shared-main
pulumi:providers:aws                                    urn:pulumi:main::shared::pulumi:providers:aws::AWS Base Provider
aws:organizations/account:Account                       urn:pulumi:main::shared::aws:organizations/account:Account::shared
pulumi:providers:aws                                    urn:pulumi:main::shared::pulumi:providers:aws::AWS Provider (shared)
aws:iam/role:Role                                       urn:pulumi:main::shared::aws:iam/role:Role::IAM Role (DeveloperAccess)
pulumi:providers:awsx                                   urn:pulumi:main::shared::pulumi:providers:awsx::default_1_0_2
pulumi:providers:pulumi                                 urn:pulumi:main::shared::pulumi:providers:pulumi::default
awsx:ec2:Vpc                                            urn:pulumi:main::shared::awsx:ec2:Vpc::Shopblocks VPC
aws:kms/key:Key                                         urn:pulumi:main::shared::aws:kms/key:Key::stack-encryption-key
aws:ec2/vpc:Vpc                                         urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc::Shopblocks VPC
aws:ec2/internetGateway:InternetGateway                 urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/internetGateway:InternetGateway::Shopblocks VPC
aws:ec2/subnet:Subnet                                   urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::Shopblocks VPC-private-1
aws:ec2/subnet:Subnet                                   urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::Shopblocks VPC-public-1
aws:ec2/routeTable:RouteTable                           urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::Shopblocks VPC-private-1
aws:ec2/eip:Eip                                         urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/eip:Eip::Shopblocks VPC-1
aws:ec2/routeTable:RouteTable                           urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::Shopblocks VPC-public-1
aws:ec2/routeTableAssociation:RouteTableAssociation     urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::Shopblocks VPC-private-1
aws:ec2/natGateway:NatGateway                           urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/natGateway:NatGateway::Shopblocks VPC-1
aws:ec2/routeTableAssociation:RouteTableAssociation     urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::Shopblocks VPC-public-1
aws:ec2/route:Route                                     urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::Shopblocks VPC-public-1
aws:ec2/route:Route                                     urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::Shopblocks VPC-private-1
aws:ec2clientvpn/endpoint:Endpoint                      urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2clientvpn/endpoint:Endpoint::Shopblocks VPN
aws:ec2clientvpn/authorizationRule:AuthorizationRule    urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2clientvpn/endpoint:Endpoint$aws:ec2clientvpn/authorizationRule:AuthorizationRule::VPN AuthorizationRule
aws:ec2clientvpn/networkAssociation:NetworkAssociation  urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2clientvpn/endpoint:Endpoint$aws:ec2clientvpn/networkAssociation:NetworkAssociation::VPN NetworkAssociation (private: subnet-0996b285ad94ad607)
aws:ec2clientvpn/authorizationRule:AuthorizationRule    urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2clientvpn/endpoint:Endpoint$aws:ec2clientvpn/authorizationRule:AuthorizationRule::VPN AuthorizationRule - ifconfig.me (34.160.111.145/32)
aws:ec2clientvpn/route:Route                            urn:pulumi:main::shared::awsx:ec2:Vpc$aws:ec2clientvpn/endpoint:Endpoint$aws:ec2clientvpn/route:Route::VPN Route - ifconfig.me - 34.160.111.145/32 (private: subnet-0996b285ad94ad607)

Found no pending operations associated with shopblocks/main

Backend
Name           pulumi.com
URL            https://app.pulumi.com/bowersbros
User           bowersbros
Organizations  bowersbros, xxx, shopblocks

Dependencies:
NAME            VERSION
@types/node     16.18.39
@pulumi/aws     5.42.0
@pulumi/awsx    1.0.2
@pulumi/pulumi  3.76.1

Pulumi locates its logs in /var/folders/0y/pwpzp9cs2nb2dqwjh7qcg0sm0000gn/T/ by default```