Azurerm provider not working with custom Azure Cloud Metadata Host (secret / private / undocumented cloud support)

laughn-man commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v1.1.7 on darwin_amd64

provider registry.terraform.io/hashicorp/azurerm v2.98.0

Affected Resource(s)

This affects the core azurerm provider.

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

https://drive.google.com/file/d/10CwuPjVP3SKWj_vuHhHqylxEHVLCqUOw/view?usp=sharing

Debug Output

https://gist.github.com/laughn-man/51824d5a77c61924215565a7b16f2746

Panic Output

No panic output

Expected Behaviour

The terraform apply command should have attempted to create the storage account normally.

Actual Behaviour

Terraform apply throws the error, unable to find environment "testa" from endpoint "localhost:7045": invalid environment specified: testa. There are 2 issues here:

Even though TestA is a profile being returned from the localhost:7045 metadata host, terraform can't seem to see it. The error message has the profile name as testa, instead of TestA but this doesn't seem to be the issue. If you do the above steps for the testb profile you will still get the error.
In main.tf the provider variable environment is set to Blah. This seems to be totally ignored, and instead azurerm is using the default cloud setup on the CLI instead. Since the metadata_host variable is set, environment should be set to the profile to use, if I understand the documentation correctly.

Steps to Reproduce

All the code for this is in the provided AzureEndpoint.zip file. It contains 2 pieces, a .Net 6 API project that will create a metadata host site on localhost, and the test Terraform project. To reproduce the issue follow the below steps:

Extract the contents of the AzureEndpoint.zip file. This should create a folder called AzureEndpoint.
In a terminal go to the extracted folder.
Run the following to build and run the the custom endpoint. .Net 6 is required.

dotnet build
dotnet run

This will create a https://localhost:7045/metadata/endpoints site that will return 2 profiles TestA and testb. You will likely have to add the self signed certificate returned from this site to the trusted certificates on your local machine, or else terraform will complain about not trusting it in the next step.

Open a second terminal and cd into the AzureEndpoint/terraform-test folder. Then do the following:

# Register a new Azure Cloud with the Azure CLI.
az cloud register --name TestA --cloud-config @"cloud.json"
# Set the TestA Cloud as default.
az cloud set --name TestA
# Login to TestA. Since it is pointed to the public cloud your public Azure account should still work.
az login
# Init terraform
terraform init
# Apply terraform, this will give you the error.
terraform apply

Important Factoids

This issue does not seem to be a problem when running in a standard Azure Cloud. It only affects custom Cloud environments. It looks like this issue was introduced in version 2.94.0. 2.93.1 does not throw the error.

References

tombuildsstuff commented 2 years ago

hi @laughn-man

Thanks for opening this issue.

Unfortunately Custom Cloud Environments aren't supported in the Azure Provider - so this appears to be working as intended at this time. So that we could better understand this use-case, could you explain this use-case a little further?

Thanks!

laughn-man commented 2 years ago

If custom cloud environments are not supported then what is the metadata_host parameter used for then? From reading the azurerm documentation it seems like it is supported:

(Optional) The Hostname of the Azure Metadata Service (for example management.azure.com), used to obtain the Cloud Environment when using a Custom Azure Environment. This can also be sourced from the ARM_METADATA_HOSTNAME

laughn-man commented 2 years ago

For my use case there are certain government Azure clouds that cannot be registered as a standard cloud with Azure. If azurerm cannot support custom clouds then we will likely have to abandon it for much of our use.

laughn-man commented 2 years ago

Hey @tombuildsstuff, I apologize, my previous comments were a little short and I didn't mean them to be. I was just surprised to hear that you are not supporting custom Azure clouds.

If you wouldn't mind please let me know reason azurerm is not longer supporting custom cloud environments (it looks like before version 2.94 that the custom environments worked), and also how the metadata_host parameter is supposed to be used.

Much Thanks

manicminer commented 2 years ago

@laughn-man I've looked into this and it seems like custom cloud envs stopped working when we introduced support for MSAL (v2) authentication tokens and Microsoft Graph. You might be aware that we are currently migrating away from v1 auth tokens and Azure Active Directory Graph, since they are both deprecated and the AAD Graph APIs are due to be discontinued mid-2022.

I've been looking to see if we can work around this, but unfortunately to properly support custom cloud environments we will need to know how to access Microsoft Graph in those environments. Currently, the AAD Graph endpoint is published by the metadata service, and I note that it's also published in the sample metadata app you provided, however MS Graph endpoints are not. Likewise, it is not currently published by the metadata service for any of the public/national clouds either.

Having the correct MS Graph endpoint published by the metadata service would be a necessary prerequisite for us to resume support for custom cloud environments. Do you know if there is an updated metadata schema available for this?

laughn-man commented 2 years ago

Hi @manicminer, thanks for getting back to me. That is interesting about the MS Graph endpoint. I can't find an updated metadata schema that has the MS Graph endpoint. It almost seems like MS wants you know the MS Graph endpoint based on the Cloud you are using, see https://docs.microsoft.com/en-us/graph/deployments#microsoft-graph-and-graph-explorer-service-root-endpoints, so I don't know if it will ever appear in the metadata.

None if this helps for custom clouds however. I will ask around a bit about it and post if I find out anything.

manicminer commented 2 years ago

@laughn-man I have no idea whether MS Graph is supported in custom clouds, anything you can find out in that regard would be a good start 👍

laughn-man commented 2 years ago

After some discussions the MS Graph endpoint is supported in custom clouds, however it will be on a custom endpoint. I think if you were to support it, you would have to add a provider argument to take the Graph endpoint. Something like graph_host, that would work similar to how metadata_host does.

tombuildsstuff commented 2 years ago

@laughn-man we've got a thread with Microsoft trying to track down if this information is available via the MetaData Endpoint, for example: https://management.azure.com/metadata/endpoints?api-version=2020-06-01

That endpoint is available in all Azure Environments, although the information returned can differ on the environment this is run within e.g.:

Since the AAD Graph endpoint is returned there, we believe the MS Graph endpoint should be returned here too (potentially it already is in a new API version) - but we'd need that information returned to be able to surface this, I'm not sure if you're aware if that's available in a new API Version?

As this Environment configuration and the rest of the Env. configuration is pulled from the MetaData Endpoint, it makes sense for this to be returned, so we'd need to wait to hear back from Microsoft here to determine how to proceed rather than adding a new field for this specifically.

zack-is-cool commented 2 years ago

@laughn-man did you ever find a solution for this?

@tombuildsstuff I also see some comments from you where it's stated that the endpoints are gathered dynamically from instance metadata, does this work with any private azure cloud that isn't in the list of clouds... but should have a metadata endpoint?

I'm not very good with go, but it looks like this is a possibility to use a combination of AZURESTACKCLOUD and AZURE_ENVIRONMENT_FILEPATH if all else fails? https://github.com/Azure/go-autorest/blob/main/autorest/azure/environments.go#L293-L326

Smgilliam25 commented 2 years ago

@tombuildsstuff Any updates here? Using a private Cloud instance and pulling Metadata URL returns: "Invalid Environment Specified" with AzureRM version 3.12.0...However through trace logs I am able to see that I'm hitting the correct endpoints and storing my statefile in Blob.

laughn-man commented 2 years ago

Hey @Actionn, I'm afraid we never found a solution for our issue. We ended up switching to bicep.

manicminer commented 2 years ago

As mentioned above, the ARM metadata endpoint needs to return a Microsoft Graph endpoint in order for us to support private clouds, as we have dropped support for the legacy AAD Graph API.

Smgilliam25 commented 2 years ago

In my case the metadata does return the Graph Endpoints...I was able to get this to function w/ Provider ARM version 2.85.0. 3.0 > Did not work, neither did 2.1

TheBlackMini commented 2 years ago

What about Azure Stack Hub? That is supposed to work using environment "stack" and metadata_host "management.azurestack.local"

But it returns Error: unable to locate metadata for environment "stack" from custom metadata host "management.azurestack.local"

tombuildsstuff commented 2 years ago

@TheBlackMini this provider does not support Azure Stack Hub - see the separate Azure Stack Provider

TheBlackMini commented 2 years ago

@tombuildsstuff Yup, right you are! My bad.

However, the confusion came from here azurerm backend configuration documentation

manicminer commented 1 year ago

Now that the Microsoft Graph endpoint is included in ARM metadata, we should be able to support this.

manicminer commented 1 year ago

With the introduction of our in-house transport and authentication packages in #20320, undocumented clouds are now supported by means of the metadata_host provider property from version v3.44.1 of the provider and later.

The custom metadata service must support the metadata schema version 2022-09-01 and must include, at a minimum, the following fields (plus endpoints and/or suffixes for any data plane services you are using):

{
  "authentication": {
    "loginEndpoint": "https://login-endpoint"
  },
  "resourceManager":"https://resourcemanager-endpoint/",
  "microsoftGraphResourceId":"https://msgraph-resource-id/"
}

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm