Closed monster-cookie closed 2 years ago
jaydesai-automate
commented
2 years ago In my case, similar behavior. Hangs for 5 mins then spits out an error. This is when you run a Plan. If you slip over to version 2.99.0 It does not give any error.
Error: retrievingcontactfor KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
nbjohnson
commented
2 years ago I also have a similar issue, but not sure if it is the same exact one. On v2.99 my keyvault and secret lookups work fine, but on any v3 my keyvault lookups work fine but any secret lookups from those vaults hang for 5-10 min and timeout with the following error:
Error: making Read request on Azure KeyVault Secret <secret name>: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: failed to acquire MSAL token
I am using CLI auth.
There was another thread where other people were having keyvault issues and seems some of the problems were fixed but mine weren't and seems like there is still an issue if the above issues are anything to go by: https://github.com/hashicorp/terraform-provider-azurerm/issues/16052
monster-cookie
commented
2 years ago In my case I'm using service principal auth for everything
tombuildsstuff
commented
2 years ago hi @monster-cookie
Thanks for opening this issue.
Taking a look through here this isn't trying to access the URI, it's trying to gain an access token for that URI, which is necessary in order to access the Key Vault within this Environment. As a part of 3.0 we've switched from using ADAL to MSAL, more information on the changes can be found here - https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-migration - and whilst the Key Vault Application should be enabled by default, it'd be worth confirming if it's registered for the Service Principal that you're using here?
The other two comments above are unrelated and are associated with #16052 - so I've marked those as off-topic, but in that case please subscribe to/keep that discussion within that issue so that we're not conflating two different issues here.
Thanks!
monster-cookie
commented
2 years ago @tombuildsstuff my problem with this is the URL https://vault.microsoftazure.us/ isn't in our state, definition files, or azure gov portal. It also doesn't appear to exist in DNS. If its a generic REST API for Azure Gov Vault then its not supported in azure gov yet which is very possible because Azure Gov lags behind Azure Public by 3-6 months and Azure DoD by 1-2 years. The URL that should be being used for those secrets data resources is https://v4secrets-global.vault.usgovcloudapi.net/.
tombuildsstuff
commented
2 years ago @monster-cookie the URI doesn't exist, it's used as the Resource when requesting a Resource Manager Authentication Token from MS Graph to access Key Vault - the returned token can then be used to access whichever key vault is necessary.
As mentioned above, you'd need to confirm if the Service Principal being used has MSAL access to Key Vault (in Azure Active Directory) - but @manicminer should be able to confirm if there's anything extra needed here?
monster-cookie
commented
2 years ago I just went through all the settings for the app and service principal, and it should have full access to the graph APIs. I've been trying to find docs for any special settings/permissions needed for MSAL but mostly it seems to match what ADAL needed. I should not with AzureRM 3.1.0 I now get the certificate error from the other ticket #16052
monster-cookie
commented
2 years ago 
sorry the image didn't upload/link
monster-cookie
commented
2 years ago Here is the updated debug logs for 3.1.0
2022-04-08T11:22:00.196-0600 [ERROR] vertex "azurerm_key_vault.v4secrets-global" error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
2022-04-08T11:22:00.196-0600 [TRACE] vertex "azurerm_key_vault.v4secrets-global": visit complete, with errors
2022-04-08T11:22:00.197-0600 [TRACE] vertex "azurerm_key_vault.v4secrets-global": dynamic subgraph encountered errors: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
2022-04-08T11:22:00.197-0600 [ERROR] vertex "azurerm_key_vault.v4secrets-global" error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
2022-04-08T11:22:00.197-0600 [TRACE] vertex "azurerm_key_vault.v4secrets-global": visit complete, with errors
2022-04-08T11:22:00.197-0600 [TRACE] vertex "azurerm_key_vault.v4secrets-global (expand)": dynamic subgraph encountered errors: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
2022-04-08T11:22:00.197-0600 [ERROR] vertex "azurerm_key_vault.v4secrets-global (expand)" error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
2022-04-08T11:22:00.197-0600 [TRACE] vertex "azurerm_key_vault.v4secrets-global (expand)": visit complete, with errors
2022-04-08T11:22:00.197-0600 [TRACE] dag/walk: upstream of "azurerm_monitor_diagnostic_setting.v4secrets-global-log-diags (expand)" errored, so skipping
adamday2
commented
2 years ago We're running into the same issue on Azure Government, running from an Azure AD account, no SPs involved.
As a point of reference, that resource URI also does not work for us with the Azure CLI. When I try to retrieve an access token for https://vault.microsoftazure.us, I get the same error as noted in the original post:
az account get-access-token --resource https://vault.microsoftazure.us
AADSTS500011: The resource principal named https://vault.microsoftazure.us was not found in the tenant named <tenant name here>. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant.But I can successfully retrieve a token from https://vault.azure.net
az account get-access-token --resource https://vault.azure.net
{
"accessToken": "<token here>",
"expiresOn": -
"subscription": -
"tenant": -
"tokenType": "Bearer"
}az cloud show | grep name
"name": "AzureUSGovernment",Is there any possibility that the vault uri is not region-specific?
monster-cookie
commented
2 years ago You can get a token from https://vault.usgovcloudapi.net also which matches the normal azure gov api end point replacements
{ "accessToken": "[REDACTED]", "expiresOn": "2022-04-08 12:42:43.000000", "subscription": "12160f2c-3952-419c-a854-c8d76d988909", "tenant": "0b50b1fa-3c16-4c29-a5cb-d4bb4d1db625", "tokenType": "Bearer" }
adamday2
commented
2 years ago This seems like it could very likely be an issue with the underlying endpoint reference. I'm not very familiar with this codebase, but I think its relying on https://github.com/manicminer/hamilton/blob/main/environments/endpoints.go#L36
If that's the case, the URI appears to be a copy/paste from Germany, with a suffix change, so it seems quite possible that this is just an incorrect endpoint being referenced for gov: https://github.com/manicminer/hamilton/commit/948b9fe8eab9a755d28697bd4396d1fe6945354e
nbjohnson
commented
2 years ago Is there any update on this issue? The above post might be the issue: https://github.com/manicminer/hamilton/blob/main/environments/endpoints.go#L36
As that repository is imported: https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/clients/builder.go#L16 and it looks like our environments are built on those endpoints: https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/clients/builder.go#L61
The usgov keyvault api endpoint listed in that repo exactly matches the endpoint that doesn't seem to work, so I think that is what needs to be updated to https://vault.usgovcloudapi.net/
manicminer
commented
2 years ago @monster-cookie, @adamday2 Thanks for the report and investigation. You are correct, the App ID URI for Vault in USGovernment is incorrect in the hamilton/environments package. We'll have a fix out for this in the next release - apologies it didn't make it into this week's release.
monster-cookie
commented
2 years ago Cool thanks for the work in getting it ready. I look forward to testing when it releases.
github-actions[bot]
commented
2 years ago This functionality has been released in v3.3.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.
For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!
monster-cookie
commented
2 years ago I can confirm this is working for me in azure gov now
github-actions[bot]
commented
2 years ago I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Is there an existing issue for this?
Community Note
Terraform Version
1.1.7
AzureRM Provider Version
3.0.2
Affected Resource(s)/Data Source(s)
azurerm_key_vault_secret
Terraform Configuration Files
Debug Output/Panic Output
Expected Behaviour
Pull and create variable for the secret
Actual Behaviour
Request fails by trying to access an invalid URL https://vault.microsoftazure.us. This URL is not in our state, definition files, or azure account. It should also be noted that the apply/plan hangs for 10-60 minutes before failing. It should also be noted this exact code works fine with 2.99.0
Steps to Reproduce
Run terraform plan or terraform apply
Important Factoids
We are in Azure Government
References
No response