search

Azure / deployment-stacks

Contains Deployment Stacks CLI scripts and releases
MIT License
89 stars 7 forks source link

Deployment Stack stuck in "Deploying" state #143

Closed cuanl-cd closed 9 months ago

cuanl-cd commented 9 months ago

Describe the bug I am testing out Subscription level Deployment Stacks. This has been working successfully but last week I accidently input the incorrect parameter for some Scheduled Query Rule resources. The Deployment itself indicates "failed" but the Deployment Stack is just stuck in the "Deploying" and has been like this since the 7th of December.

I have tried to remove it or update via Azure Pipelines and Powershell but it just states:

<Deployment Stack Name> could not be deployed as it is already in a non-terminal state 'Deploying'.

To Reproduce Steps to reproduce the behavior:

  1. Go to Subscription
  2. Click on Deployment Stacks
  3. Observe Deployment Stack stuck in Deploying State

Expected behavior The deployment should time out and/or fail eventually so that the Deployment Stack can either be updated with the correct values or Deleted

Screenshots If applicable, add screenshots to help explain your problem.

Repro Environment Host OS: Powershell Version:

Server Debugging Information Correlation ID: Tenant ID: Timestamp of issue (please include time zone): Data Center (eg, West Central US, West Europe):

Additional context Add any other context about the problem here.

dantedallag commented 9 months ago

@cuanl-cd Could you please provide us a correlation id so we can look into this further?

cuanl-cd commented 9 months ago

@dantedallag

Please see some correlation ID's below for failed Deployment Stack delete from yesterday:

f36dbc98-52da-4866-a592-5497d765d25f fb1803a9-baba-42e0-b788-51c67bcbfc72

ramondegoede commented 9 months ago

Hi @dantedallag , Just wanted to chime in here. We also have some issues regarding deployment stacks which is stuck in a state since december the 7th. which could be related to this issue as well. Our state which it is stuck on is "initiated". We have this with 3 of our deployment stacks out of the +- 40 we deploy, so it doesn't seem to be consistent.

Here some of the related correlation ID's to hopefully help out: cf41e85b-8c51-4305-b0f2-da97eae8c921 14fb0d61-bf74-49c2-9ea9-ad74b0528d4f ae9fd48a-8a14-4c5c-8281-2250342c01da

dantedallag commented 9 months ago

@cuanl-cd and @ramondegoede, we seem to have pinpointed the issue that have caused both of your stacks to get stuck in this non-terminal state. Basically, there was a time window on December 7th where some stack operations were not able to fully complete. A fix is currently being rolled out to prevent something like this from happening in the future.

To both of you and anyone else who has stacks stuck in a similar state: We can reset them on our side. Please provide us with resource ids for all stacks needing to be reset here: ARMDeploymentStacks@microsoft.com. Thanks!

cuanl-cd commented 9 months ago

Thank you @dantedallag.

I reached out to the provided email address and my Deployment Stacks were then marked as Failed allowing me to delete them.

As such, I will close off this issue

pjelar commented 8 months ago

I'm faced with the same issue. This is a new occurrence that happened on the 28th of January 2024.

Correlation id: 7bf2bef5-3377-44f9-8f4d-e8cefc381258