Elasticsearch deployment automation does not work anymore

[thesse1]

Calling

PUT https://management.azure.com/subscriptions/309065ca-a060-4592-8096-b74694126b61/resourceGroups/azure-demo-01-monitoring-rg/providers/Microsoft.Elastic/monitors/test-elasticsearch?api-version=2023-02-01-preview
{"location":"westeurope","properties":{"monitoringStatus":"Enabled","userInfo":{"emailAddress":"THES@softwareag.com"}},"sku":{"name":"ess-consumption-2024_Monthly@TIDgmz7xq9ge3py"},"tags":{}}

does not work anymore for a token retrieved locally using az account get-access-token. I am always getting the following error message:

Status 400
{
    "error": {
        "code": "BadRequest",
        "message": "Cannot proceed with the request as the user is not authorized"
    }
}

It is still working fine with a token retrieved in Azure Cloud Shell (in the Azure Portal) - for the same user.

Some more details:

I have been using Terraform for setting up my Azure infrastructure including an instance of Managed Elasticsearch for several months, and it has always been working fine until yesterday (08.12.23) morning CET. It failed for the first time yesterday (08.12.23) afternoon, and it has failed ever since.

Please find the Terraform configuration of a repro case in the attachment.

Since Friday afternoon, it would always complain:

│ Error: creating Monitor (Subscription: "309065ca-a060-4592-8096-b74694126b61"
│ Resource Group Name: "azure-demo-01-monitoring-rg"
│ Monitor Name: "azure-demo-01-elasticsearch"): performing MonitorsCreate: unexpected status 400 with error: BadRequest: Cannot proceed with the request as the user is not authorized
│
│   with module.monitoring.azurerm_elastic_cloud_elasticsearch.default,
│   on monitoring\monitoring-elasticsearch.tf line 1, in resource "azurerm_elastic_cloud_elasticsearch" "default":
│    1: resource "azurerm_elastic_cloud_elasticsearch" "default" {

Yes, I am the owner of the resource group, and Terraform is working fine for dozens of other resources. Yes, I can log-in to Elastic Cloud with my Microsoft account THES@softwareag.com. Yes, I can create an Elasticsearch instance in the same RG with the same resource configuration with the same user in Azure portal.

I have tried the Terraform script with location westeurope, eastus and southeastasia. Same result.

I have tried creating the resource using Azure CLI:

az elastic monitor create -n test-elasticsearch -g azure-demo-01-monitoring-rg --user-info "{firstName:Thomas,lastName:Hesse,companyName:'Software AG',emailAddress:THES@softwareag.com}" --sku "{name:ess-consumption-2024_Monthly@TIDgmz7xq9ge3py}"

Result:

(BadRequest) Cannot proceed with the request as the user is not authorized
Code: BadRequest
Message: Cannot proceed with the request as the user is not authorized

I have exported an ARM template in the Azure Portal, cf. attachment. I can create the resource using the template in the Azure Portal, but it fails when I try the following:

az deployment group create --resource-group azure-demo-01-monitoring-rg --template-file ExportedTemplate-azure-demo-01-elasticsearch.json --parameters @ExportedTemplate-azure-demo-01-elasticsearch-parameters.json

Result:

{"status":"Failed","error":{"code":"DeploymentFailed","target":"/subscriptions/309065ca-a060-4592-8096-b74694126b61/resourceGroups/azure-demo-01-monitoring-rg/providers/Microsoft.Resources/deployments/ExportedTemplate-azure-demo-01-elasticsearch","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"BadRequest","message":"Cannot proceed with the request as the user is not authorized"}]}}

I have even tried calling the Azure Management API:

PUT https://management.azure.com/subscriptions/309065ca-a060-4592-8096-b74694126b61/resourceGroups/azure-demo-01-monitoring-rg/providers/Microsoft.Elastic/monitors/test-elasticsearch?api-version=2023-02-01-preview
{"location":"westeurope","properties":{"monitoringStatus":"Enabled","userInfo":{"emailAddress":"THES@softwareag.com"}},"sku":{"name":"ess-consumption-2024_Monthly@TIDgmz7xq9ge3py"},"tags":{}}

Result:

Status 400
{
    "error": {
        "code": "BadRequest",
        "message": "Cannot proceed with the request as the user is not authorized"
    }
}

I have tried multiple versions of the API.

Yes, I am using a valid token, the API is working fine for other resource types.

Some more results of my analysis:

• Terraform is working fine when I run it in an Azure DevOps pipeline. It only fails when I run it locally.
• az elastic monitor list is working fine. Only az elastic monitor create fails when I run it locally.
• az elastic monitor create is working fine when I run it in a Cloud Shell in Azure Portal, but not on my PC.
• Creating a token in Cloud Shell with az account get-access-token and using it in Postman on my PC is working fine, but not with a token created with az account get-access-token on my PC. When I compare the tokens, I see a difference in the unique_name claim. It is THES@softwareag.com for the token created in Cloud Shell and thomas.hesse@softwareag.com for the token created locally, cf. attachments.

Maybe this is causing the issue? Please note that my Elastic Cloud username is THES@softwareag.com. Are you taking the Elastic Cloud username from the unique_name claim of the token and not from the userInfo.emailAddress of the request body? Or are you forwarding the token to Elastic Cloud? (I hope not…)

Anyhow: How can I log-in to Azure CLI in such a way that Terraform and az elastic monitor create are working locally again?

JWT_Payload_Local.json JWT_Payload_Cloud_Shell.json

Currently I see no way of setting up the resource automatically. Please help!

Best regards, Thomas

[JackTn]

@poshett can you check on this issue? Thanks. Let me know if you have any question!

[RocketRaccoon]

Same issue, looks like replacing elastic_cloud_email_address with the current account's email do the trick but as OP mentioned it breaks the idea of automated provisioning.