search

microsoft / azure-container-apps

Roadmap and issues for Azure Container Apps
MIT License
356 stars 27 forks source link

unable to create a custom dns suffix with azapi #1082

Open jason-berk-k1x opened 4 months ago

jason-berk-k1x commented 4 months ago

Please provide us with the following information:

This issue is a: (mark with an x)

Issue description

I have an existing Container App Environment (ie CAE). I want to configure a "Custom DNS suffix", which is currently in Preview. I used the AzureRM provider to create the Log Analytics Workspace and the CAE (among other resources). I now want to use the AzApi provider to update the existing CAE to enable my Custom DNS suffix

Steps to reproduce

  1. create a log analytics workspace with the azurerm_log_analytics_workspace resource
  2. create a container env with the azurerm_container_app_environment resource
  3. create the DNS A and TXT records (to verify your domain)
  4. use the azapi_update_resource to update the CAE resource and add a Custom DNS Suffix

Expected behavior [What you expected to happen.]

DNS Suffix is successfully added and apps in the CAE are now accessible at app-name.sub.domain.io

Actual behavior [What actually happened.]

resource update fails with

│ Error: creating/updating "Resource: (ResourceId \"/subscriptions/my-sub/resourceGroups/rg-dev/providers/Microsoft.App/managedEnvironments/aca-env-dev\" / Api Version \"2023-05-02-preview\")": PUT https://management.azure.com/subscriptions/my-sub/resourceGroups/rg-dev/providers/Microsoft.App/managedEnvironments/aca-env-dev │ -------------------------------------------------------------------------------- │ RESPONSE 400: 400 Bad Request │ ERROR CODE: InvalidRequestParameterWithDetails │ -------------------------------------------------------------------------------- │ { │ "error": { │ "code": "InvalidRequestParameterWithDetails", │ "message": "LogAnalyticsConfiguration is invalid. Must provide a valid LogAnalyticsConfiguration" │ } │ } │ --------------------------------------------------------------------------------

Additional context

jason-berk-k1x commented 4 months ago

after spending way too much time trying to figure all this nonsense out, I appear to have finally gotten it working. The azapi update appears to require the appLogsConfiguration. Based on #990 I noticed the call is doing a PUT and maybe if it did a PATCH it would work?????

anyway, after a lot of head banging...... this appears to work and enable my custom dns suffix:

resource "azapi_update_resource" "cae_custom_dns_suffix" {
  type = "Microsoft.App/managedEnvironments@2023-05-02-preview"
  resource_id = azurerm_container_app_environment.container_app_env.id

  body = jsonencode({
    properties = {
      customDomainConfiguration = {
        dnsSuffix = "sub.domain.io" // use your value here
        certificatePassword = "blah blah" // use your value here
        certificateValue = filebase64("./ssl-cert.pfx") // use your value here
      },
      appLogsConfiguration = {
        destination = "log-analytics"
        logAnalyticsConfiguration = {
          customerId = azurerm_log_analytics_workspace.cae_logs.workspace_id
          sharedKey = azurerm_log_analytics_workspace.cae_logs.primary_shared_key
        }
      }
    }
  })
}

now I'm going to go update it to pull my cert bytes and password from a key vault..... assuming I get that working, I'll post my complete TF.... wish me luck

jason-berk-k1x commented 4 months ago

spent two days trying to get this to work.......to no avail......

using the config in my previous post, everything works perfectly. But we don't run TF from our local machines, so I don't want to refer to a local file via filebase64("./ssl-cert.pfx")

I loaded my certificate into a key vault and tried to read it from there:

# get a handle to the cert in the key vault
data "azurerm_key_vault_certificate" "ssl_cert" {
  name               = "cert-name"
  key_vault_id   = data.azurerm_key_vault.my-vault.id
}

# use AzApi to update the env
resource "azapi_update_resource" "cae_custom_dns_suffix" {
  type = "Microsoft.App/managedEnvironments@2023-05-02-preview"
  resource_id = azurerm_container_app_environment.container_app_env.id

  body = jsonencode({
    properties = {
      customDomainConfiguration = {
        dnsSuffix = "my.domain.io"
        certificatePassword = "my-cert-passphrase" # I can always load this value from a secret later
        # instead of reading the cert from a local file, which I can't really do in CI.....
        # certificateValue = filebase64("wildcard.pfx")
        certificateValue = data.azurerm_key_vault_certificate.ssl_cert.certificate_data_base64
      },
      appLogsConfiguration = {
        destination = "log-analytics"
        logAnalyticsConfiguration = {
          customerId = azurerm_log_analytics_workspace.cae_logs.workspace_id
          sharedKey = azurerm_log_analytics_workspace.cae_logs.primary_shared_key
        }
      }
    }
  })

  // don't attempt to enable the custom domain config until the DNS records are added
  depends_on = [
    null_resource.wait_for_dns_prop
  ]
}

which fails with

│ -------------------------------------------------------------------------------- │ RESPONSE 400: 400 Bad Request │ ERROR CODE: CertificateMustContainOnePrivateKey │ -------------------------------------------------------------------------------- │ { │ "error": { │ "code": "CertificateMustContainOnePrivateKey", │ "message": "Certificate must contain one private key." │ } │ } │ --------------------------------------------------------------------------------

if I instead use

certificateValue = data.azurerm_key_vault_certificate.ssl_cert.certificate_data

I get

│ -------------------------------------------------------------------------------- │ RESPONSE 400: 400 Bad Request │ ERROR CODE: InvalidCertificateValueFormat │ -------------------------------------------------------------------------------- │ { │ "error": { │ "code": "InvalidCertificateValueFormat", │ "message": "The certificate value should be base64 string." │ } │ } │ --------------------------------------------------------------------------------

soooooooo.....

  1. How exactly am I supposed to provide a base64 encoded certificate that contains a private key?
  2. why does the exact same PFX file work when read from the local file system but fail when read from a vault?

how on earth are people using ACA...... every time I turn around there's another bug / road block to automating things with TF / AzAPI

jason-berk-k1x commented 4 months ago

ok...... so here's what I did.....

I base64 encoded the PFX file on my local machine:

base64 -i my-cert.pfx | pbcopy

then I manually created a secret in my vault named certificate and saved the base64 encoded value and updated my TF like so:

data "azurerm_key_vault_secret" "cert" {
  name               = "certificate"
  key_vault_id   = data.azurerm_key_vault.my-vault.id
}

data "azurerm_key_vault_secret" "cert_pass" {
  name               = "cert-passphrase"
  key_vault_id   = data.azurerm_key_vault.my-vault.id
}

/*
enable the custom DNS suffix
technically, I should have to only set "customDomainConfiguration", but
that doesn't work: https://github.com/microsoft/azure-container-apps/issues/1082
*/
resource "azapi_update_resource" "cae_custom_dns_suffix" {
  type = "Microsoft.App/managedEnvironments@2023-05-02-preview"
  resource_id = azurerm_container_app_environment.container_app_env.id

  body = jsonencode({
    properties = {
      customDomainConfiguration = {
        dnsSuffix = "my.domain.io"
        certificatePassword = data.azurerm_key_vault_secret.cert_pass.value
        certificateValue = data.azurerm_key_vault_secret.cert.value
      },
      appLogsConfiguration = {
        destination = "log-analytics"
        logAnalyticsConfiguration = {
          customerId = azurerm_log_analytics_workspace.cae_logs.workspace_id
          sharedKey = azurerm_log_analytics_workspace.cae_logs.primary_shared_key
        }
      }
    }
  })

  // don't attempt to enable the custom domain config until the DNS records are added
  depends_on = [
    null_resource.wait_for_dns_prop
  ]
}

this appears to work and be completely automated in TF..... Hopefully someone (or my future self) finds all this helpful.

KoenR3 commented 4 months ago

This is a bug that was not there and is now messing up my workflows. key of the log analytics is null in the capp JSON, this messes up every update action as the shared key is required.

jason-berk-k1x commented 4 months ago

so from what I can gather, when you put a cert in a vault, if that cert has a private key attached, it gets stripped. Or maybe when you ask the vault for the cert the returned bytes stripped the private key. That would explain the CertificateMustContainOnePrivateKey error. I guess you just can't use certs in a vault with ACA yet.....

KoenR3 commented 4 months ago

No it is not related to the private key, the log analytics shared key is null. You can see it in the json view of the container app environment. If you update only the values for the certificates it will fail validation because the shared key is null. I am currently resolving it by reapplying the loganalytics settings as well

KoenR3 commented 4 months ago

so from what I can gather, when you put a cert in a vault, if that cert has a private key attached, it gets stripped. Or maybe when you ask the vault for the cert the returned bytes stripped the private key. That would explain the CertificateMustContainOnePrivateKey error. I guess you just can't use certs in a vault with ACA yet.....

If you put a certificate in a key vault, the actual key is stored seperately. If you want to build up the certificate, you need to join them back together, like this:

     certificateValue = base64encode(join(
          "",
          [
            data.azurerm_key_vault_certificate_data.CERT-CAPPS-INTERNAL.key,
            "-----BEGIN CERTIFICATE-----",
            split("-----BEGIN CERTIFICATE-----", data.azurerm_key_vault_certificate_data.CERT-CAPPS-INTERNAL.pem)[data.azurerm_key_vault_certificate_data.CERT-CAPPS-INTERNAL.certificates_count]
          ]
        ))

This uses a letsencrypt certificate that will remove the certificate chain and only combine the private key back with the actual certificate

lgmorand commented 3 months ago

@anthonychu two of my Cx are also complaining about this issue. Do you know if we already have an internal bug for his one and if someone is working on it please ? thanks :)

jason-berk-k1x commented 2 weeks ago

just for clarity, I was able to use a newer AzureRM provider version that supported certificates. I got this all working purely in TF w/o needing AzApi. That said, I still put the certificate and it's key in a vault as base64 encoded secrets.

when I create the CAE, I can do this:

resource "azurerm_container_app_environment_certificate" "cert" {
  name                         = "my-cert-name"
  container_app_environment_id = azurerm_container_app_environment.cae.id
  certificate_blob_base64      = data.azurerm_key_vault_secret.cert.value
  certificate_password         = data.azurerm_key_vault_secret.cert_passphrase.value
}

and when I create my ACA App, I can do this (along with many other things, like setting up DNS)

/*
create the custom domain for the ACA app using the certificate from the CAE
*/
resource "azurerm_container_app_custom_domain" "custom_domain" {
  for_each = local.cname_record_names

  name                                     = "${each.value}.${data.azurerm_dns_zone.zone.name}"
  container_app_id                         = azurerm_container_app.app.id
  container_app_environment_certificate_id = data.azurerm_container_app_environment_certificate.cert.id
  certificate_binding_type                 = "SniEnabled"

  depends_on = [
    null_resource.wait_for_txt_record_propagation,
    null_resource.wait_for_cname_record_propagation
  ]
}
lgmorand commented 2 weeks ago

@jason-berk-k1x for the cert in KV, do you use a specific format ? pfx, etc ? My Cx is trying to do exactly this but everytime the RM says the cert is incorrect. thanks :)

jason-berk-k1x commented 2 weeks ago

pretty sure it's a PEM cert that's been base64 encoded. I'd recommend getting it all working in the portal, then tearing it down and doing in TF