az network application-gateway root-cert create fails with keyvault url

sanjaydebnath commented 3 years ago

Describe the bug

Command Name az network application-gateway root-cert create

Errors:


azure.core.pipeline.policies._universal : {
  "status": "Failed",
  "error": {
    "code": "ApplicationGatewayKeyVaultSecretException",
    "message": "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/<removed>/resourceGroups/MC_<removed>-aks-test_westus2/providers/Microsoft.Network/applicationGateways/<removed>appgty-test'. See details below:",
    "details": [
      {
        "code": "ApplicationGatewayTrustedRootCertificateInvalidData",
        "message": "Data for certificate /subscriptions/<removed>/resourceGroups/MC_<removed>-aks-test_westus2/providers/Microsoft.Network/applicationGateways/<removed>-appgty-test/trustedRootCertificates/<removed>backendtls is invalid."
      }
    ]
  }
}

  <removed>File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-0cx4jf6i\azure\core\polling\base_polling.py", line 500, in run
azure.core.exceptions.HttpResponseError: (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/<removed>/resourceGroups/MC_<removed>_westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below:

cli.azure.cli.core.azclierror : (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/<removed>/resourceGroups/MC_<removed>_westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below:
(ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/<removed>/resourceGroups/MC_<removed>_westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below:
Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x035F3DB0>]
az_command_data_logger : exit code: 1
Command ran in 14.222 seconds (init: 0.230, invoke: 13.993)
telemetry.save : Save telemetry record of length 3321 in cach

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Create KV, add two certs with autorotation on for frontend & backend ssl in RG
Create AKS cluster with AGIC using this
Create an identity separately in a different resource group to access KV certs & assign the same as App Gty identity too
AGIC identity has "Managed Identity Operator" role on this new identity
create ssl cert with az network application-gateway ssl-cert create -n frontend-tls --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME --key-vault-secret-id $unversionedSecretId
TRY to create root cert using same az network application-gateway root-cert create --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME --name backend-tls --keyvault-secret $unversionedSecretId2
The secret ids look like 'https://kvName.vault.azure.net/secrets/backendcert' & 'https://kvName.vault.azure.net/secrets/frontendendcert'
Expected Behavior

Should work fine & pull the cert from KV. Surprisingly the frontend cert works but the backend one fails to be created. I dont want to manually upload the cert as that breaks the auto rotation ideally.

Environment Summary
```
Windows-10-10.0.19041-SP0
Python 3.6.8
Installer: MSI
```

azure-cli 2.17.1 *

Extensions: account 0.2.1 aks-preview 0.4.72 application-insights 0.1.13 kusto 0.2.0


## Additional Context
Identity assigned to App Gty:
`
az network application-gateway identity show --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME
{
  "principalId": null,
  "tenantId": null,
  "type": "userAssigned",
  "userAssignedIdentities": {
    "/subscriptions/{removed}/resourcegroups/test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/appgty-ssl-identity-test": {
      "clientId": "ebb81e58-{removed}",
      "principalId": "5d09d269-{removed}"
    }
  }
}
`
KV Access:
`appgty-ssl-identity-test` has access to get & list for all 3 secret, cert & key. And KV is set for all network access. Also to note the cert in KV is provided by a private CA authority. 
<!--Please don't remove this:-->
<!--auto-generated-->

yonzhan commented 3 years ago

network

sanjaydebnath commented 3 years ago

Ok my certificate in AKV for root-cert is a chain certificate. I see that the app gty root cert has to be cer file for the root certificate of my chain cert. Is there a different way of enabling auto rotation for root-cert in that case.... rather than manually uploading? My app gty is WAF_v2.

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

Issue Details

## Describe the bug **Command Name** `az network application-gateway root-cert create` **Errors:** ``` azure.core.pipeline.policies._universal : { "status": "Failed", "error": { "code": "ApplicationGatewayKeyVaultSecretException", "message": "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC_-aks-test_westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below:", "details": [ { "code": "ApplicationGatewayTrustedRootCertificateInvalidData", "message": "Data for certificate /subscriptions//resourceGroups/MC_-aks-test_westus2/providers/Microsoft.Network/applicationGateways/-appgty-test/trustedRootCertificates/backendtls is invalid." } ] } } File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-0cx4jf6i\azure\core\polling\base_polling.py", line 500, in run azure.core.exceptions.HttpResponseError: (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC__westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below: cli.azure.cli.core.azclierror : (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC__westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below: (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC__westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below: Event: Cli.PostExecute [] az_command_data_logger : exit code: 1 Command ran in 14.222 seconds (init: 0.230, invoke: 13.993) telemetry.save : Save telemetry record of length 3321 in cach ``` ## To Reproduce: Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information. - Create KV, add two certs with autorotation on for frontend & backend ssl in RG - Create AKS cluster with AGIC using [this](https://azure.github.io/application-gateway-kubernetes-ingress/setup/install-new-windows-cluster/) - Create an identity separately in a different resource group to access KV certs & assign the same as App Gty identity too - AGIC identity has "Managed Identity Operator" role on this new identity - create ssl cert with `az network application-gateway ssl-cert create -n frontend-tls --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME --key-vault-secret-id $unversionedSecretId` - TRY to create root cert using same `az network application-gateway root-cert create --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME --name backend-tls --keyvault-secret $unversionedSecretId2` - The secret ids look like 'https://kvName.vault.azure.net/secrets/backendcert' & 'https://kvName.vault.azure.net/secrets/frontendendcert' ## Expected Behavior Should work fine & pull the cert from KV. **Surprisingly the frontend cert works but the backend one fails to be created**. I dont want to manually upload the cert as that breaks the auto rotation ideally. ## Environment Summary ``` Windows-10-10.0.19041-SP0 Python 3.6.8 Installer: MSI azure-cli 2.17.1 * Extensions: account 0.2.1 aks-preview 0.4.72 application-insights 0.1.13 kusto 0.2.0 ``` ## Additional Context Identity assigned to App Gty: ` az network application-gateway identity show --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME { "principalId": null, "tenantId": null, "type": "userAssigned", "userAssignedIdentities": { "/subscriptions/{removed}/resourcegroups/test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/appgty-ssl-identity-test": { "clientId": "ebb81e58-{removed}", "principalId": "5d09d269-{removed}" } } } ` KV Access: `appgty-ssl-identity-test` has access to get & list for all 3 secret, cert & key. And KV is set for all network access. Also to note the cert in KV is provided by a private CA authority.

Author:	sanjaydebnath
Assignees:	msyyc
Labels:	`Network`, `OKR3.2 Candidate`, `Service Attention`, `feature-request`
Milestone:	S185

msyyc commented 3 years ago

Hi@service team, according to the description, it seems need support from service to allow this kind of certificate.

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @appgwsuppgithub.

Issue Details

## Describe the bug **Command Name** `az network application-gateway root-cert create` **Errors:** ``` azure.core.pipeline.policies._universal : { "status": "Failed", "error": { "code": "ApplicationGatewayKeyVaultSecretException", "message": "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC_-aks-test_westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below:", "details": [ { "code": "ApplicationGatewayTrustedRootCertificateInvalidData", "message": "Data for certificate /subscriptions//resourceGroups/MC_-aks-test_westus2/providers/Microsoft.Network/applicationGateways/-appgty-test/trustedRootCertificates/backendtls is invalid." } ] } } File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-0cx4jf6i\azure\core\polling\base_polling.py", line 500, in run azure.core.exceptions.HttpResponseError: (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC__westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below: cli.azure.cli.core.azclierror : (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC__westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below: (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC__westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below: Event: Cli.PostExecute [] az_command_data_logger : exit code: 1 Command ran in 14.222 seconds (init: 0.230, invoke: 13.993) telemetry.save : Save telemetry record of length 3321 in cach ``` ## To Reproduce: Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information. - Create KV, add two certs with autorotation on for frontend & backend ssl in RG - Create AKS cluster with AGIC using [this](https://azure.github.io/application-gateway-kubernetes-ingress/setup/install-new-windows-cluster/) - Create an identity separately in a different resource group to access KV certs & assign the same as App Gty identity too - AGIC identity has "Managed Identity Operator" role on this new identity - create ssl cert with `az network application-gateway ssl-cert create -n frontend-tls --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME --key-vault-secret-id $unversionedSecretId` - TRY to create root cert using same `az network application-gateway root-cert create --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME --name backend-tls --keyvault-secret $unversionedSecretId2` - The secret ids look like 'https://kvName.vault.azure.net/secrets/backendcert' & 'https://kvName.vault.azure.net/secrets/frontendendcert' ## Expected Behavior Should work fine & pull the cert from KV. **Surprisingly the frontend cert works but the backend one fails to be created**. I dont want to manually upload the cert as that breaks the auto rotation ideally. ## Environment Summary ``` Windows-10-10.0.19041-SP0 Python 3.6.8 Installer: MSI azure-cli 2.17.1 * Extensions: account 0.2.1 aks-preview 0.4.72 application-insights 0.1.13 kusto 0.2.0 ``` ## Additional Context Identity assigned to App Gty: ` az network application-gateway identity show --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME { "principalId": null, "tenantId": null, "type": "userAssigned", "userAssignedIdentities": { "/subscriptions/{removed}/resourcegroups/test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/appgty-ssl-identity-test": { "clientId": "ebb81e58-{removed}", "principalId": "5d09d269-{removed}" } } } ` KV Access: `appgty-ssl-identity-test` has access to get & list for all 3 secret, cert & key. And KV is set for all network access. Also to note the cert in KV is provided by a private CA authority.

Author:	sanjaydebnath
Assignees:	msyyc
Labels:	`Network - Application Gateway`, `Service Attention`
Milestone:	Backlog

Phrow commented 3 years ago

Hi, I'm having the same issue. Adding regular ssl-certs from key vault works, but adding a root-cert does not. Maybe it's related to the fact that the root-cert cannot be added as "certificate" to key vault (because there is no key), but instead must be added as "secret"? In any case, the only way I found for me was to add the root cert to the gateway by direct upload instead of a key vault reference, which I consider an intermediate "workaround", but not a soluion.

adin3d commented 2 years ago

same problem - the az cli has the --keyvault secret argument for az network application-gateway root-cert create command but I get Code: ApplicationGatewayTrustedRootCertificateInvalidData

If I use the file directly instead of the secret in keyvault it works

And I have to get it from a keyvault secret cause apparently you can't add trusted root certificates in Keyvaul certificates

So not sure how the format is once you have it the keyvault secret but I imagine something is not ok there

ghost commented 2 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

Issue Details

## Describe the bug **Command Name** `az network application-gateway root-cert create` **Errors:** ``` azure.core.pipeline.policies._universal : { "status": "Failed", "error": { "code": "ApplicationGatewayKeyVaultSecretException", "message": "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC_-aks-test_westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below:", "details": [ { "code": "ApplicationGatewayTrustedRootCertificateInvalidData", "message": "Data for certificate /subscriptions//resourceGroups/MC_-aks-test_westus2/providers/Microsoft.Network/applicationGateways/-appgty-test/trustedRootCertificates/backendtls is invalid." } ] } } File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-0cx4jf6i\azure\core\polling\base_polling.py", line 500, in run azure.core.exceptions.HttpResponseError: (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC__westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below: cli.azure.cli.core.azclierror : (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC__westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below: (ApplicationGatewayKeyVaultSecretException) Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions//resourceGroups/MC__westus2/providers/Microsoft.Network/applicationGateways/appgty-test'. See details below: Event: Cli.PostExecute [] az_command_data_logger : exit code: 1 Command ran in 14.222 seconds (init: 0.230, invoke: 13.993) telemetry.save : Save telemetry record of length 3321 in cach ``` ## To Reproduce: Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information. - Create KV, add two certs with autorotation on for frontend & backend ssl in RG - Create AKS cluster with AGIC using [this](https://azure.github.io/application-gateway-kubernetes-ingress/setup/install-new-windows-cluster/) - Create an identity separately in a different resource group to access KV certs & assign the same as App Gty identity too - AGIC identity has "Managed Identity Operator" role on this new identity - create ssl cert with `az network application-gateway ssl-cert create -n frontend-tls --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME --key-vault-secret-id $unversionedSecretId` - TRY to create root cert using same `az network application-gateway root-cert create --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME --name backend-tls --keyvault-secret $unversionedSecretId2` - The secret ids look like 'https://kvName.vault.azure.net/secrets/backendcert' & 'https://kvName.vault.azure.net/secrets/frontendendcert' ## Expected Behavior Should work fine & pull the cert from KV. **Surprisingly the frontend cert works but the backend one fails to be created**. I dont want to manually upload the cert as that breaks the auto rotation ideally. ## Environment Summary ``` Windows-10-10.0.19041-SP0 Python 3.6.8 Installer: MSI azure-cli 2.17.1 * Extensions: account 0.2.1 aks-preview 0.4.72 application-insights 0.1.13 kusto 0.2.0 ``` ## Additional Context Identity assigned to App Gty: ` az network application-gateway identity show --gateway-name $APP_GTY_NAME -g $AKS_RG_NAME { "principalId": null, "tenantId": null, "type": "userAssigned", "userAssignedIdentities": { "/subscriptions/{removed}/resourcegroups/test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/appgty-ssl-identity-test": { "clientId": "ebb81e58-{removed}", "principalId": "5d09d269-{removed}" } } } ` KV Access: `appgty-ssl-identity-test` has access to get & list for all 3 secret, cert & key. And KV is set for all network access. Also to note the cert in KV is provided by a private CA authority.

Author:	sanjaydebnath
Assignees:	kairu-ms
Labels:	`Network`, `Service Attention`, `Network - Application Gateway`
Milestone:	-

pmontanaro commented 2 years ago

I can confirm I'm having this issue too.

I'm able to upload .cer root certificates via clickOps on my App Gateway but cannot automate my deployment from key vault, resulting in:

{ "code": "ApplicationGatewayTrustedRootCertificateInvalidData", "message": "Data for certificate /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Network/applicationGateways/<app_gateway_name>/trustedRootCertificates/<certname> is invalid." }

rahul-presidio commented 2 years ago

Facing the above issue when the root certificate is referenced from the keyvault

{
"code": "ApplicationGatewayTrustedRootCertificateInvalidData",
"message": "Data for certificate <full-resource-id>trustedRootCertificates/root-cert is invalid."
}

larwood commented 2 years ago

Hi All,

Today I successfully added a root-cert to an Application Gateway (WAF V2) from a secret stored in a Key Vault.

Key Vault

Add the secret to the key vault. az keyvault secret set --name Internal-CA-Chain-base64 --vault-name My-KeyVault --subscription My-Sub --file Internal-CA-Chain.cer --encoding base64

Internal-CA-Chain.cer is a PEM file with Root and two Intermediates like so:

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

In order like so: Issuer: RootCA Subject: RootCA Issuer: RootCA Subject: InterCA Issuer: InterCA Subject: IssuingCA

Once the secret is created retrieve the Secret ID.

$ az keyvault secret list --vault-name My-KeyVault --subscription My-Sub --output json --query "[].{Name:name,ID:id}"
[
  {
    "ID": "https://my-keyvault.vault.azure.net/secrets/Internal-CA-Chain-base64",
    "Name": "Internal-CA-Chain-base64"
  }
]

Application Gateway

Add the root-cert to the application gateway. az network application-gateway root-cert create --keyvault-secret 'https://my-keyvault.vault.azure.net/secrets/Internal-CA-Chain-base64' --name Internal-CA-Chain --gateway-name My-AppGW --resource-group My-ResGroup --subscription My-Sub

List the root-certs installed to the application gateway.

az network application-gateway root-cert list --gateway-name My-AppGW --resource-group My-ResGroup --subscription My-Sub --output json --query "[].{Name:name,ProvisioningState:provisioningState,KeyVaultSecretId:keyVaultSecretId}"
[
  {
    "KeyVaultSecretId": "https://my-keyvault.vault.azure.net/secrets/Internal-CA-Chain-base64",
    "Name": "Internal-CA-Chain",
    "ProvisioningState": "Succeeded"
  }
]

toddmacintyre commented 10 months ago

I'm still facing this despite trying the above steps. Any updates for this issue?

Azure / azure-cli