Isuue with "make letsencrypt": `Error creating new order :: too many certificates already issued for "eastus.cloudapp.azure.com". Retry after 2024-04-25T20:00:00Z:`

[BiologyGeek]

Hello team,

Describe the bug When attempting to run make letsencrypt after initial deployment, the following error message is displayed:

Waiting for network rule to take effect
Created network rule on storage account
Checking for index.html file in storage account
index.html already present
Saving debug log to /workspaces/AzureTRE-Deployment/letsencrypt/logs/letsencrypt.log
Account registered.
Requesting a certificate for mytre.eastus.cloudapp.azure.com
An unexpected error occurred:
Error creating new order :: too many certificates already issued for "eastus.cloudapp.azure.com". Retry after 2024-04-25T20:00:00Z: see https://letsencrypt.org/docs/rate-limits/
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /workspaces/AzureTRE-Deployment/letsencrypt/logs/letsencrypt.log or re-run Certbot with -v for more details.
make: *** [/home/vscode/AzureTRE/Makefile:119: letsencrypt] Error 1

Error creating new order :: too many certificates already issued for "eastus.cloudapp.azure.com". Retry after 2024-04-25T20:00:00Z:

Steps to reproduce

1. Run make letsencrypt for an instance deployed in the East US region.

Azure TRE release version: 0.17

---

Question: Before addressing this issue programmatically, is there a way to resolve it manually?

[tim-allen-ck]

Hi @BiologyGeek looks like there's a limit of 50 Certs per base domain (eastus.cloudapp.azure.com) per week. https://letsencrypt.org/docs/rate-limits/ and its asking you to wait to the 25th to try again. Could you try creating in a different location?

[BiologyGeek]

Thanks you @tim-allen-ck, The Let's Encrypt documentation you provided states:

The main limit is Certificates per Registered Domain (50 per week). A registered domain is, generally speaking, the part of the domain you purchased from your domain name registrar. For instance, in the name www.example.com, the registered domain is example.com. In new.blog.example.co.uk, the registered domain is example.co.uk.

Based on this document, I believe 'azure.com' is considered the Registered Domain to which rate limits are applied, not 'eastus.cloudapp.azure.com'. Is my understanding correct? In this case, if I deploy in another region, would 'azure.com' still be the Registered Domain, with only a component like 'eastus' being replaced in this URL: eastus.cloudapp.azure.com?

[tim-allen-ck]

Thanks you @tim-allen-ck, The Let's Encrypt documentation you provided states:

The main limit is Certificates per Registered Domain (50 per week). A registered domain is, generally speaking, the part of the domain you purchased from your domain name registrar. For instance, in the name www.example.com, the registered domain is example.com. In new.blog.example.co.uk, the registered domain is example.co.uk.

Based on this document, I believe 'azure.com' is considered the Registered Domain to which rate limits are applied, not 'eastus.cloudapp.azure.com'. Is my understanding correct? In this case, if I deploy in another region, would 'azure.com' still be the Registered Domain, with only a component like 'eastus' being replaced in this URL: eastus.cloudapp.azure.com?

Ahh yes sorry @BiologyGeek, you're right the Registered domain is azure.com. Yes it will still be the registered domain if deploying to a new region.

[tim-allen-ck]

@BiologyGeek to get round the issue for now try added in --test-cert here file to use Lets Encrypt staging which has higher limits.

[tim-allen-ck]

1540

[BiologyGeek]

@BiologyGeek to get round the issue for now try added in --test-cert here file to use Lets Encrypt staging which has higher limits.

Thank you @tim-allen-ck!

Here is the result:

Successfully received certificate.
Certificate is saved at: /workspaces/AzureTRE-Deployment/letsencrypt/live/mytre.eastus.cloudapp.azure.com/fullchain.pem
Key is saved at:         /workspaces/AzureTRE-Deployment/letsencrypt/live/mytre.eastus.cloudapp.azure.com/privkey.pem
This certificate expires on 2024-07-24.
These files will be updated when the certificate renews.

However, after running the command curl https://<my_azure_tre_fqdn>/api/health, I encounter the following error message:

curl: (60) schannel: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted.
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Also, the browser still shows a warning indicator and states 'Not Secure'.

Will this issue impact the functionality of shared services or other components? Or can I proceed without worrying about further issues?

---

Update: During the deployment of the Nexus service via web UI, I encountered this error:

Deployment failed. There was an issue with the latest deployment or update for this resource. Please see the Operations panel within the resource for details.

The log shows entries like this:

Error message: Unable to find image 'mytre.azurecr.io/tre-shared-service-certs@sha<Hash was Here>' locally Saving debug log to /cnab/app/letsencrypt/logs/letsencrypt.log An unexpected error occurred: Error creating new order :: too many certificates already issued for "eastus.cloudapp.azure.com". Retry after 2024-04-25T20:00:00Z: see https://letsencrypt.org/docs/rate-limits/ Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /cnab/app/letsencrypt/logs/letsencrypt.log or re-run Certbot with -v for more details. error running command /cnab/app /bin/bash ./scripts/letsencrypt.sh --application_gateway_name ******* --cert_name nexus-ssl --fqdn ******* --keyvault_name ******* --resource_group_name ******* --storage_account_name *******: exit status 1 Error: error running command /cnab/app /bin/bash ./scripts/letsencrypt.sh --application_gateway_name ******* --cert_name nexus-ssl --fqdn ******* --keyvault_name ******* --resource_group_name ******* --storage_account_name *******: exit status 1 

[tim-allen-ck]

ahh that message from the nexus is from the cert shared service. /templates/shared_services/certs/scripts/letsencrypt.sh this file will also need the --test-cert adding

[marrobi]

Just a heads up, the staging certs aren't trusted by default, so if don't want warnings need to add the root cert to the trusted store on the browser. It's fine for development, but for production, best using a custom domain name on the app gateway(s).

[BiologyGeek]

ahh that message from the nexus is from the cert shared service. /templates/shared_services/certs/scripts/letsencrypt.sh this file will also need the --test-cert adding

Thank you @tim-allen-ck! I tried to do this, but I encountered this error:

Error message: Saving debug log to /cnab/app/letsencrypt/logs/letsencrypt.log An unexpected error occurred: Error creating new order :: too many certificates already issued for "eastus.cloudapp.azure.com". Retry after 2024-04-26T14:00:00Z: see https://letsencrypt.org/docs/rate-limits/ Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /cnab/app/letsencrypt/logs/letsencrypt.log or re-run Certbot with -v for more details. error running command /cnab/app /bin/bash ./scripts/letsencrypt.sh -- application_gateway_name ******* -- cert_name nexus-ssl -- fgdn ******* __keyvault_name ******* -- resource_group_name ******* -- storage_account_name *******: exit status 1 Error: error running command /cnab/app /bin/bash***.storage_account_name *******: exit status 1

---

Just a heads up, the staging certs aren't trusted by default, so if don't want warnings need to add the root cert to the trusted store on the browser. It's fine for development, but for production, best using a custom domain name on the app gateway(s).

Thank you @marrobi! I couldn't find step-by-step instructions on adding a custom domain within the TRE documentation. Could you please point me to some documents or resources that guide me on how to do so?

[marrobi]

It's something we need do some more work on as part of #1540.

As you can see here - https://github.com/microsoft/AzureTRE/blob/cb59c99233575b096207e6afc288af9da7ee0256/core/terraform/appgateway/certificate.tf#L15C1-L64C2 a certificate called letsencrypt is created in KeyKault. You can replace that with a cert for your DNS name, and point a custom domain name to the App Gateway public IP.

I'll have to have a think about Nexus, as it's not something I've personally done.

[tim-allen-ck]

Closing, will be looked at in #1540