search

Azure / deployment-stacks

Contains Deployment Stacks CLI scripts and releases
MIT License
89 stars 7 forks source link

Jun 2024 HEADS UP: HTTP 500 when creating first deployment stack #168

Open snarkywolverine opened 3 months ago

snarkywolverine commented 3 months ago

According to our telemetry, a few customers have hit an HTTP 500 when attempting to create their first deployment stack.

This issue occurs when a customer is creating their first deployment stack, and has also never used deployment scripts in the past. This occurs when the customer's first stack creation also specifies a DenySetting other than "none".

While we work on rolling out a fix, we recommend deploying the stack first with DenySetting = None, and then re-deploying with the actual, desired DenySetting value.

Note that this is only true for the very first Deployment stack -- subsequent stacks can be created with any DenySetting value that is desired without issue.

Sorry for the inconvenience! Feel free to reach out if you have additional questions.

Parth099 commented 1 month ago

Commenting on this to keep it active. Issue still effecting us as of August 8

snarkywolverine commented 1 month ago

@Parth099 Yes, we are still working on the fix. Were you able to use the workaround provided above?

aslev commented 1 month ago

Hello, I am commenting here on request at a support engineer from Microsoft.

I experienced these problems but it behaved a bit differently than described in this issue. They occured between the 12th and 13th of August. During the 12th, I deployed maybe 30-50 times. I tried with DenySetting to None, which DID work, however when trying to deploy again with any other setting, it failed. I tried to deploy with None, then change that stack with both az cli, powershell and azure portal, all failed. I removed it and deployed a new one, with both same and a different name, all failed. I tried many different combinations, all failed.

During the 13th, it was still not working so I tried to deploy to another region than westeurope. I tried westus. This deploy worked 100% instantly with all settings on DenySetting. However, when trying again on westeurope, it still failed.

Then I gave up but I tried it again later that same day. And now, deploying to westeurope worked just as intended and it has been working since.

This seem to me that were some kind of error or block. But it was not removed instantly when I deployed with DenySetting = None. It took maybe 36 hours before it started working.

As a consultant who works with many clients, deployment stacks seems very useful. But it is going to be a annoyance if this is gonna happen every time we want to start working with it on a new client. Let me know if I can help with logs or deploy scripts that I used.

Parth099 commented 1 month ago

@Parth099 Yes, we are still working on the fix. Were you able to use the workaround provided above?

Hi, yes we were able to deploy using none for the first time. Let me know if you guys need additional data to fix this for others.

dantedallag commented 4 weeks ago

Hi @aslev, could you create a separate issue with appropriate correlation ids (if you have them) so we can look into what happened here?