hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.6k stars 4.64k forks source link

`azurerm_container_app_custom_domain` fails parsing the certificate ID for managed certificates #25788

Closed ghjklw closed 1 month ago

ghjklw commented 6 months ago

Is there an existing issue for this?

Community Note

Terraform Version

1.8.1

AzureRM Provider Version

3.101.0

Affected Resource(s)/Data Source(s)

azurerm_container_app_custom_domain

Terraform Configuration Files

resource "azurerm_container_app_custom_domain" "airflow" {
  container_app_id = module.container_app["webserver"].app.id
  name             = trimsuffix(azurerm_dns_cname_record.airflow.fqdn, ".")
  # The FQDN of the DNS CNAME Record which has a full-stop at the end is by design.
  # We must therefore remove it

  depends_on = [azurerm_dns_txt_record.airflow_verification]
  lifecycle {
    ignore_changes = [certificate_binding_type, container_app_environment_certificate_id]
  }
}

Debug Output/Panic Output

│ Error: parsing "/subscriptions/<subscription_id>/resourceGroups/<rg_name>/providers/Microsoft.App/managedEnvironments/<cae_name>/managedCertificates/<certificate_name>": parsing segment "staticCertificates": parsing the Certificate ID: the segment at position 8 didn't match
│ 
│ Expected a Certificate ID that matched:
│ 
│ > /subscriptions/12345678-1234-9876-4563-123456789012/resourceGroups/example-resource-group/providers/Microsoft.App/managedEnvironments/managedEnvironmentValue/certificates/certificateValue
│ 
│ However this value was provided:
│ 
│ > /subscriptions/<subscription_id>/resourceGroups/<rg_name>/providers/Microsoft.App/managedEnvironments/<cae_name>/managedCertificates/<certificate_name>
│ 
│ The parsed Resource ID was missing a value for the segment at position 8
│ (which should be the literal value "certificates").

Expected Behaviour

I would expect the managed certificate ID to be ignored, as defined in ignore_changes... or to be correctly parsed by accepting either certificates or managedCertificates. In that case, when running terraform apply I owuld have expected to get Apply complete! Resources: 0 added, 0 changed, 0 destroyed. as the container app was already in the expected state.

As a side note, when the domain was initially added, the certificate was not generated and the domain wasn't bound (certificate_binding_type remains set to Disabled). I don't know if that's expected behaviour, but it's slightly unfortunate as it doesn't enable fully automated deployment.

Actual Behaviour

When running terraform apply the first time, the custom domain name was deployed.

I then triggered the certificate generation and bound it manually.

When I ran terraform apply again, instead of getting Apply complete! Resources: 0 added, 0 changed, 0 destroyed. as expected, I got the error message above.

Steps to Reproduce

  1. Run terraform apply to create a new container app with a custom domain name
  2. Check that the domain name is created
  3. Manually trigger the creation of a managed certificate and bind it in Azure Portal
  4. Run terraform apply again

Important Factoids

No response

References

This is a newly added resource: #25356

sinbai commented 6 months ago

Hi @ghjklw thanks for opening this issue. Per the information provided, I assume that the error is reported by resource azurerm_container_app_environment_certificate, correct? If so, I assume that this is by design because the Rest API corresponding to "azurerm_container_app_environment_certificate" is Certificates, and manually trigger the creation of a managed certificate and bind it in Azure Portal in step3, the created managed certificate corresponding Rest API is ManagedCertificates.As far as I know, currently, Terraform does not have resources to support the ManagedCertificates API.

ghjklw commented 6 months ago

Hi @sinbai,

Thanks for looking into it! No I am not using this resource at all, only azurerm_container_app_custom_domain. If I'm not mistaken, the whole point of #25356 was to be able to use Azure Managed Certificates, and I did apply ignore_changes = [certificate_binding_type, container_app_environment_certificate_id] as explicitly mentioned in the documentation.

CRidge commented 6 months ago

Hi @ghjklw and @sinbai,

I'm seeing the same issue. It would be great to know if the manual step of creating the managed certificate is by design, or if it is supposed to be created automatically on the first deployment - and if so, if it is normal for it to take some time?

dhduvall commented 6 months ago

I'll note that the test added in #25356 only checks for the existence of the domain, not whether the certificate is set up properly.

fabrideci commented 6 months ago

I confirm the behaviour described above, in opposite to what's written in the documentation:

"Omit this value (container_app_environment_certificate_id) if you wish to use an Azure Managed certificate. You must create the relevant DNS verification steps before this process will be successful."

The managed certificate does never get created by Terraform, even though all DNS verification steps are run before.

It's useless having to generate the certificate through the portal afterwards in a fully automated infra-as-code environment as where Terraform is supposed to help with.

brett-matson commented 5 months ago

I'm getting this error too. Does anyone have a work around, other than not using managed certificates?

fabrideci commented 5 months ago

I'm getting this error too. Does anyone have a work around, other than not using managed certificates?

Our workaround is using the AzAPI provider to create both the Container App and Managed Certificate. Unfortunately, due to technical constraints on both the actual Azure API and the AzAPI provider itself, we must:

  1. Create the Container App first, with Custom Domain/s configured and bound to temporary self-signed certificate/s
  2. Create the Managed Certificate/s (requires Custom Domain/s to be there in advance - see point 1)
  3. Update the Container App to edit the Custom Domain/s and bind them to the Managed Certificate/s.

Note that azapi_update_resource requires to pass the entire JSON body of the Container App configuration again, so it won't definitely be nice as you'll have duplicated code to maintain.

brett-matson commented 5 months ago

@fabrideci Yikes, thanks for that. This isn't ideal.

LynnAU commented 5 months ago

I've had success using a local-exec provisioner to run a bash script which calls out to azcli to provision the managed cert and bind it to the domain for me.

To bind the custom domain to the container app, use the following command: az containerapp hostname add -g <resource_group> -n <container_app_name> --hostname <custom_domain> This will bind the hostname to the app without requiring a certificate to do so.

Then to create a managed certificate: az containerapp env certificate create -g <resource_group> -n <container_app_env_name> --hostname <custom_domain> --validation-method CNAME You can change the validation method to your preference.

And finally to bind the cert to the domain (providing the domain is already bound to the container app): az containerapp hostname bind -g <resource_group> -n <container_app_name> --hostname <custom_domain> --environment <container_app_env_name> --certificate <certificate_name>

I've set my script up to work for adding/removing custom domains on already deployed container apps, here's a link to the gist if anyone is curious: https://gist.github.com/LynnAU/131426847d2793c76e36548f9937f966 The nice part about the script is, it will bind the custom domain to the container app if it's not already, import an existing managed certificate if one exists, and exit if the managed certificate is already bound.

brett-matson commented 5 months ago

@LynnAU Wow, thanks. That's very helpful.

sinbai commented 5 months ago

Hi @ghjklw I have submitted PR to fix this issue. Could you please track it for more updates?

rodyvansambeek commented 5 months ago

This is a blocking issue for me too. I used a manual binding using the portal and now my Terraform scripts are failing. Is there a workaround available to be able to run the scripts, while using the manual binding to the custom domain?

Edited -> After trying out the manual "az containerapp" way described by @LynnAU to add the certificate and binding, it works. Somehow terraform does not recognizes that new certificate and binding as something to destroy on the next "terraform apply".

1TT-Chris commented 5 months ago

Thanks to @LynnAU for the awesome script. I had to make some tweaks to get it working. The final command to bind the hostname and certificate failed for me. Running it with --debug revealed that it fails because, when only specifying a certificate name, the Azure CLI only looks in the "certificates" resource namespace when managed certificates live in "managedCertificates".

The workaround was to use the managed certificate resource ID, rather than the name.

I've left a note on the gist, and raised a bug for the CLI issue here: https://github.com/Azure/azure-cli/issues/29119

thomasdewulf commented 4 months ago

I've been having the same issue, the script provided by @LynnAU works, but when the state gets refreshed on subsequent runs the error still occurs. Any ideas?

resource "azurerm_container_app_custom_domain" "app_domain" {
  container_app_id = azurerm_container_app.web_app.id
  name = "my.domain.com"
  provisioner "local-exec" {
    command = "chmod +x ./scripts/bind-custom-domain.sh; ./scripts/bind-custom-domain.sh" # Based on working dir set in github action
    environment = {
      RESOURCE_GROUP = data.azurerm_resource_group.resource_group.name
      CONTAINER_APP_ENV_NAME = azurerm_container_app_environment.container_app_environment.name
      CUSTOM_DOMAIN = "my.domain.com"
      CONTAINER_APP_NAME = azurerm_container_app.web_app.name
    }
  }
}
bolsteryorick commented 4 months ago

@thomasdewulf For this workaround you need to forgo using the azurerm_container_app_custom_domain resource entirely and use a null_resource resource to execute the local-exec in. If you want it to also destroy the domain on a terraform destroy you will have to add a script that does that with a when argument of 'destroy'

LynnAU commented 4 months ago

Glad you liked the script guys, I'm back with some updates.

I've tweaked my create script to add some more checks prior to provisioning a certificate and included a destroy script for cleanup. The gist now also contains some terraform files to execute the scripts for you, just need to place the main.tf and variables.tf files in a folder and use it as a custom module in your deployments.

I've added a DNS check using dig to verify the asuid TXT record for the custom domain exists before main execution, this is because we use Cloudflare to handle our DNS and when adding custom domains after an already provisioned Container App, the execution of the script in our deployment pipeline happens within seconds of updating the DNS records (so I needed to delay my script for propagation). Feel free to comment out that section if it's not to your liking or needs, hell do what you want with the script lol...

@thomasdewulf bolsteryorick is correct, you need to use a null_resource terraform resource for the scripts to work correctly, check my gist linked above if you want to copy what I have. The files in the gist is what my module contains for provisioning custom domains.

@1TT-Chris Cheers for leaving feedback, weird that you had to change those lines. I wonder if it's to do with the AZCLI version I use, which is 2.53.0 currently. I've added this as a caveat to the gist's README incase anybody else encounters the problem.

fabrideci commented 2 months ago

Is there any update on this? Still it's not possible to use azurerm_container_app_custom_domain with managed certificates after months of being reported.

github-actions[bot] commented 2 weeks ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.