search

microsoft / AzureTRE

An accelerator to help organizations build Trusted Research Environments on Azure.
https://microsoft.github.io/AzureTRE
MIT License
169 stars 133 forks source link

Core deployment errors - Error: reading blob properties for Storage Account #4004

Open jonnyry opened 1 week ago

jonnyry commented 1 week ago

Deploying a fresh TRE from 2 commits behind main (1ffb09baf37f4599adfd65b4259fdda7564da408)...

Experienced storage account failures on two seperate deployments:

Capture

Capture3

tim-allen-ck commented 1 week ago

could be an issue with the terraform providers?

jonnyry commented 1 week ago

Just pulled your tf provider update and giving it a try now.

jonnyry commented 1 week ago

Reployed fine with the tf provider update - I'll close this off.

jonnyry commented 1 week ago

Unfortuntely just happened again on the latest codebase (including the tf provider update):

image

tim-allen-ck commented 1 week ago

how weird, and the storage account doesn't exist in the portal?

jonnyry commented 1 week ago

I have already destroyed that environment already, but I've just seen it happen again and I've checked and the storage account does exist:

image

image

jonnyry commented 1 week ago

Looks like it could be this: https://github.com/hashicorp/terraform-provider-azurerm/issues/22992

jonnyry commented 1 week ago

And for the issue to be resolved, wait for the following PR: https://github.com/hashicorp/terraform-provider-azurerm/pull/26218

jonnyry commented 1 week ago

still occurring on almost every deployment:

image

marrobi commented 1 week ago

Are you using the same TRE ID between deployments? Wonder if some sort of DNS caching going on?

jonnyry commented 1 week ago

I am yes. I think it is to do with DNS caching.

However not sure why the issue has only started occuring this week, as been reusing TRE IDs for a long time.

marrobi commented 1 week ago

Are these jobs running on a github agent? Is it a GitHub hosted one or an agent you manage?

jonnyry commented 1 week ago

Just a standard GitHub Actions Runner - not one that we manage.

marrobi commented 1 week ago

Hmm, we never really deploy, destroy, and deploy with the same ID. Wonder if you left it a couple of days before reusing the same ID.

Might be need to create an issue in the Terraform provider repository/wait for the one you mentioned, not sure its an Azure TRE issue, or something we are going to be able to resolve.

jonnyry commented 1 week ago

It's intermittent - I think it does depend on how long it was since the last deploy.

Curiously for the issue I was having this morning, I deleted the storage account that was causing the 404, re-ran the build and it completed sucessfully.

I'll hopefully manage to work around it for now and keep an eye on release of the terraform PR above.