certificateregistration.AppServiceCertificateOrderCertificate never ended is action

[stawen]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

Context (Environment)

- Azure France Central
- @pulumi/azure: 4.26.0
- @pulumi/azure-native: 1.45.0,
- @pulumi/pulumi: 3.17.1,
- @types/node@14.17.21

I want to create a public certificate with the "App Service Certificate" managed service, and link it to a Keyvault so that the generated certificate is stored in the keyvault.

Azure Official documentation : https://docs.microsoft.com/en-us/azure/app-service/configure-ssl-certificate#start-certificate-order

Pulumi Documentation : https://www.pulumi.com/registry/packages/azure-native/api-docs/certificateregistration/

The complete steps for App Service Certificate are:

• Creation an Certificate Order,
• Link with a Keyvault,
• Make a Domain verification with a key in the root domain

Here no need to go to the domain verification to reproduce my issue

I Make this test :

• When I make the request through the portal, everything is fine.
• When I do the request through Azure Rest APIs, everything is fine
• When I do it in pulumi, pulumi never gives back when the Certificate Order is associated with the Keyvault. However, the Azure listed action was completed in seconds, and seems to be ok.

The problem is that pulumi does not finish is action although the link is made

I have to kill the process, and therefore I have a synchronization problem between Azure and pulumi.

Steps to reproduce

1. The Pulumi Typescript code

Here i create :

• ressource group : rg-test-app-serv-cert
• the Keyvault with Acces Policy like the documentation say : kv-test-asc
  • no network restriction
  • To be sure, i allow me all acces to keyvault
• An AppServiceCertificateOrder (its the App Service Certificate)

import * as pulumi from '@pulumi/pulumi'
import * as resources from '@pulumi/azure-native/resources'
import * as keyvault from '@pulumi/azure-native/keyvault'
import { keyvault as keyvaultEnums } from '@pulumi/azure-native/types/enums'
import * as azuread from '@pulumi/azuread'
import * as certificateregistration from '@pulumi/azure-native/certificateregistration'

const azureConfig = new pulumi.Config('azure')
export const tenantId = azureConfig.require('tenantId')

// Common ressource group
const rgName = `rg-test-app-serv-cert`
export const rg = new resources.ResourceGroup(rgName, {
    resourceGroupName: rgName,
})

//MANDATORY DO NOT CHANGE
// ADD Microsoft.Azure.CertificateRegistration acces to kv
const certificateRegistration = azuread.getServicePrincipal({
    displayName: 'Microsoft.Azure.CertificateRegistration',
})
// Not sure mandatory, App Service can get a cert into this kv
const azureAppServiceRegistration = azuread.getServicePrincipal({
    displayName: 'Microsoft Azure App Service',
})

// Keyvault creating with Acces policy
const kv = new keyvault.Vault('kv-test-asc', {
    vaultName: 'kv-test-asc',
    resourceGroupName: rg.name,
    properties: {
        accessPolicies: [
            {
                objectId: certificateRegistration.then((sp) => sp.id),
                permissions: {
                    secrets: [keyvaultEnums.SecretPermissions.All],
                    certificates: [keyvaultEnums.CertificatePermissions.All],
                },
                tenantId: tenantId,
            },
            {
                objectId: azureAppServiceRegistration.then((sp) => sp.id),
                permissions: {
                    secrets: [keyvaultEnums.SecretPermissions.Get],
                    certificates: [keyvaultEnums.CertificatePermissions.Get],
                },
                tenantId: tenantId,
            },
            {
                objectId: 'xxxxxx-xxxxx-xxxxx-xxxxxx-xxxxxxxx', // My User Object ID
                permissions: {
                    certificates: [keyvaultEnums.CertificatePermissions.All],
                    keys: [keyvaultEnums.KeyPermissions.All],
                    secrets: [keyvaultEnums.SecretPermissions.All],
                },
                tenantId: tenantId,
            },
        ],
        enabledForDeployment: true,
        enabledForDiskEncryption: true,
        enabledForTemplateDeployment: true,
        enableSoftDelete: true,
        enablePurgeProtection: true,
        enableRbacAuthorization: false,

        sku: {
            family: keyvaultEnums.SkuFamily.A,
            name: keyvaultEnums.SkuName.Standard,
        },
        tenantId,
    },
})

const subdomain = 'my'
const fqdn = `${subdomain}.domain.io`
const formatedFqdn = `${subdomain}-domain-io`

// App Service Certificat creation
const certOrder = `cert-order-${formatedFqdn}`

const order = new certificateregistration.AppServiceCertificateOrder(
    certOrder,
    {
        certificateOrderName: certOrder,
        resourceGroupName: rg.name,
        productType: certificateregistration.CertificateProductType.StandardDomainValidatedSsl,
        distinguishedName: `CN=${fqdn}`,
        autoRenew: true,
        location: 'global',
    },
    {
        ignoreChanges: ['certificates'],
        deleteBeforeReplace: true,
    },
)

// App Service Certificat Order link with Keyvault
// Action seem to be ok in Azure side, but in Pulumi side the action never ends, we must force the stop. The backend stack is desync with azure, and in
// ci/cd pipeline it's not acceptable.
const linkCertOrder = `link-kv-cert-order-${formatedFqdn}`

const register = new certificateregistration.AppServiceCertificateOrderCertificate(
    linkCertOrder,
    {
        name: formatedFqdn,
        certificateOrderName: order.name,
        resourceGroupName: rg.name,
        keyVaultId: kv.id,
        keyVaultSecretName: formatedFqdn,
    },
    {
        dependsOn: [order],
        customTimeouts: { create: '5m' }, //5 minutes max, because Azure Rest API responds in seconds, so there is no need to wait too long.
    },
)

export const certificat = {
    certOrder: order,
    certRegister: register,
}

I have here a time out on AppServiceCertificateOrderCertificate at the end of 5 minutes, because I'm bored. If I don't put it on, the action doesn't end even after 60 minutes

2. Test with Azure Rest API

I make a Powershell script to test it I am using the keyvault created by my pulumi script below. This shows that the keyvault conf looks correct

Reference API

• Create Order : https://docs.microsoft.com/en-us/rest/api/appservice/app-service-certificate-orders/create-or-update
• Link : https://docs.microsoft.com/en-us/rest/api/appservice/app-service-certificate-orders/create-or-update-certificate

Replace $subscriptionId and $tenantId with your own

# Log in first with Connect-AzAccount if not using Cloud Shell
$subscriptionId='xxxx-xxxx-xxx-xxxx-xxxxxx'
$tenantId='xxxx-xxxx-xxx-xxxx-xxxxxx'

Select-AzSubscription -SubscriptionId $subscriptionId -Tenant $tenantId
$azContext = Get-AzContext
$azProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile
$profileClient = New-Object -TypeName Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient -ArgumentList ($azProfile)
$token = $profileClient.AcquireAccessToken($azContext.Subscription.TenantId)

$authHeader = @{
    'Content-Type'='application/json'
    'Authorization'='Bearer ' + $token.AccessToken
}

# Keyvault is already build w/ Acces Policy
$resourceGroupName='rg-test-app-serv-cert'
$subdomain="myapp"
$fqdn="${subdomain}.domain.io"
$formatedFqdn="${subdomain}-domain-io"
$certificateOrderName="cert-order-${formatedFqdn}"

# Create Certificate Order
# DOC API : https://docs.microsoft.com/en-us/rest/api/appservice/app-service-certificate-orders/create-or-update
$restUri = "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.CertificateRegistration/certificateOrders/${certificateOrderName}?api-version=2015-08-01"

$body = @{
    location='global'
    tags=@{
        "owner"='stawen'
    }
    properties=@{
        "productType"="StandardDomainValidatedSsl"
        "distinguishedName"="CN=${fqdn}"
    }
}
$json = $body | ConvertTo-Json

Invoke-WebRequest $restUri -Method Put -Body $json -ContentType 'application/json' -Headers $authHeader
# http return StatusCode : 201 - OK

# Link Cert Order with a Keyvault
# DOC API : https://docs.microsoft.com/en-us/rest/api/appservice/app-service-certificate-orders/create-or-update-certificate
$certName=$formatedFqdn
$restUri = "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.CertificateRegistration/certificateOrders/${certificateOrderName}/certificates/${certName}?api-version=2015-08-01"

$body = @{
    location='global'
    tags=@{
        "owner"='stawen'
    }
    properties=@{
        "keyVaultId"="/subscriptions/xxxx-xxxx-xxx-xxxx-xxxxxx/resourceGroups/rg-test-app-serv-cert/providers/Microsoft.KeyVault/vaults/kv-test-asc"
        "keyVaultSecretName"="${formatedFqdn}"
    }
}
$json = $body | ConvertTo-Json

Invoke-WebRequest $restUri -Method Put -Body $json -ContentType 'application/json' -Headers $authHeader
# http return StatusCode : 201 - Created - OK

# the App Service Certificat and the Link between App Service Certificat and Keyvault is OK

Everything is fine, the link action only took a few seconds and everything seems ok on the Azure side (return code 201)

It seems that the problem is on the side of the Azure-native provider and not the Rest Azure APIs, in your opinion ?

[mikhailshilkov]

Thank you @stawen for an incredibly detailed description! May I ask you to run the resource creation with debug options, log everything that happens (that will include raw HTTP payloads) and send them to me at mikhail@pulumi.com?

An example command:

pulumi up --debug -v=9 --skip-preview --yes --logflow --logtostderr

[stawen]

hi @mikhailshilkov, i send you the output file

[mikhailshilkov]

@stawen Thank you for sharing the logs with me. I opened an issue upstream in https://github.com/Azure/azure-rest-api-specs/issues/16804

We'll try to add a manual work around in the provider to get users unblocked until this is fixed on the service side (if ever).

[stawen]

@mikhailshilkov ok, thank you ! glad that can help

[ghost]

Hello @mikhailshilkov, is there any update regarding the work around? I am facing the very same issue.

[mikhailshilkov]

@agi1clj Unfortunately, I don't think we landed a workaround for this. cc @danielrbradley and @stack72 who are working on the provider these days.

[ghost]

Hi @danielrbradley / @stack72, maybe as a temporary solution, you can just put a try catch block and if exception is timed out or something to just pass. The problem is that the resource (OrderCertificate) is created, but Pulumi in pipeline fails because of the time out. But I think should be enough to just log there that there was a timed out but the resource is there.

[stawen]

Hy Everyone, I test again with the latest Azure API version for App Service Certificate and the problem is still there. So I developed a Pulumi Dynamic Provider. It allows you to create an App Service Certificate and store it into a KeyVault.

You can read the documentation in my repo: https://github.com/stawen/azure-certificate

Yo use it :

npm install @stawen/azure-certificate

I hope this helps you all.

cc @mikhailshilkov / @agi1clj / @thomas11 / @abhinav / @danielrbradley / @stack72 / @ArcasGabriel / @Christoba and @nicolas-vgl-mf

[thomas11]

Hey @stawen, that's really cool! Thank you for sharing!

It's unfortunate that MS is taking forever to triage the issue we filed.