unexpected state: ready - possible to pass CA certificate?

[ffly90]

What is your question?

I am trying to use certmagic with a private acme compatible CA. Currently I am struggling to get a certificate issued with the following command:

root@host ~ # ./acme-test 
2021/12/14 15:45:31 hostname.net: obtaining certificate: hostname.net] Obtain: hostname.net] finalizing order https://acme.myca.net:443/acme/ws/Acme.svc/order/_-1n546ZnCna12d:  unexpected state: ready - order already finalized (ca=https://acme.myca.net/acme/ws/Acme.svc/InternalWebserverACME/directory)

I have worked with this CA using several other acme clients like cert-manager and acme.sh. Those clients worked fine, although I needed to provide them with the certificate of the private ACME CA. Is there a way to pass the certificate to certmagic?

What have you already tried?

My code so far for testing:

package main

import (
    "log"
    "net/http"

    "github.com/caddyserver/certmagic"
)

func main() {
    domains := []string{"hostname.net"}
    CA := "https://acme.myca.ne/acme/ws/Acme.svc/InternalWebserverACME/directory"
    Email := "somemail@somemail.net"

    certmagic.DefaultACME.Agreed = true
    certmagic.DefaultACME.Email = Email
    certmagic.DefaultACME.CA = CA

    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, encrypted world!"))
    })
    if err := certmagic.HTTPS(domains, handler); err != nil {
        log.Fatal(err)
    }
}

Edit: FYI: some of the values are changed for security reasons.

[ffly90]

I tried with a plane caddy server and got the following:

2021/12/15 14:52:33.724 INFO    using provided configuration    {"config_file": "Caddyfile", "config_adapter": ""}
2021/12/15 14:52:33.725 WARN    input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "Caddyfile", "line": 5}
2021/12/15 14:52:33.726 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/12/15 14:52:33.727 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc000427ea0"}
2021/12/15 14:52:33.727 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2021/12/15 14:52:33.728 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2021/12/15 14:52:33.729 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2021/12/15 14:52:33.729 INFO    tls     finished cleaning storage units
2021/12/15 14:52:33.729 INFO    http    enabling automatic TLS certificate management   {"domains": ["hostname.net"]}
2021/12/15 14:52:33.730 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2021/12/15 14:52:33.730 INFO    serving initial configuration
2021/12/15 14:52:33.730 INFO    tls.obtain      acquiring lock  {"identifier": "hostname.net"}
2021/12/15 14:52:33.731 INFO    tls.obtain      lock acquired   {"identifier": "hostname.net"}
2021/12/15 14:52:34.435 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["hostname.net"], "ca": "https://acme.myca.net/acme/ws/Acme.svc/InternalWebserverACME/directory", "account": "mymail@mymail.net"}
2021/12/15 14:52:34.435 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["hostname.net"], "ca": "https://acme.myca.net/acme/ws/Acme.svc/InternalWebserverACME/directory", "account": "mymail@mymail.net"}
2021/12/15 14:52:35.008 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "hostname.net", "challenge_type": "http-01", "ca": "https://acme.myca.net/acme/ws/Acme.svc/InternalWebserverACME/directory"}
2021/12/15 14:52:35.319 INFO    tls.issuance.acme       served key authentication       {"identifier": "hostname.net", "challenge": "http-01", "remote": "192.168.165.150:56773", "distributed": false}
2021/12/15 14:52:37.460 INFO    tls.issuance.acme.acme_client   validations succeeded; finalizing order {"order": "https://acme.myca.net:443/acme/ws/Acme.svc/order/ttSvhdJFxaVxU-o"}
2021/12/15 14:52:37.855 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "hostname.net", "issuer": "acme.myca.net-acme-ws-Acme.svc-InternalWebserverACME-directory", "error": "[hostname.net] finalizing o
rder https://acme.myca.net:443/acme/ws/Acme.svc/order/ttSvhdJFxaVxU-o: unexpected state: ready - order already finalized (ca=https://acme.myca.net/acme/ws/Acme.svc/InternalWebserverACME/directory)"}
2021/12/15 14:52:37.855 ERROR   tls.obtain      will retry      {"error": "[hostname.net] Obtain: [hostname.net] finalizing order https://acme.myca.net:443/acme/ws/Acme.svc/order/ttSvhdJFxaVxU-o: unexpected state: ready - order alre
ady finalized (ca=https://acme.myca.net/acme/ws/Acme.svc/InternalWebserverACME/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 4.12377922, "max_duration": 2592000}
2021/12/15 14:53:38.726 INFO    tls.issuance.acme.acme_client   validations succeeded; finalizing order {"order": "https://acme.myca.net:443/acme/ws/Acme.svc/order/AhSI5IRoiMRO2jZ"}
2021/12/15 14:53:39.160 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "hostname.net", "issuer": "acme.myca.net-acme-ws-Acme.svc-InternalWebserverACME-directory", "error": "[hostname.net] finalizing o
rder https://acme.myca.net:443/acme/ws/Acme.svc/order/AhSI5IRoiMRO2jZ: unexpected state: ready - order already finalized (ca=https://acme.myca.net/acme/ws/Acme.svc/InternalWebserverACME/directory)"}
2021/12/15 14:53:39.160 ERROR   tls.obtain      will retry      {"error": "[hostname.net] Obtain: [hostname.net] finalizing order https://acme.myca.net:443/acme/ws/Acme.svc/order/AhSI5IRoiMRO2jZ: unexpected state: ready - order alre
ady finalized (ca=https://acme.myca.net/acme/ws/Acme.svc/InternalWebserverACME/directory)", "attempt": 2, "retrying_in": 120, "elapsed": 65.429414628, "max_duration": 2592000}

In the Caddyfile I provided the path to them pem file of the CA. Sadly the problem persists and I do not know where it is coming from. It probably doesn't connect to certmatgic directly, since it also happens with caddy. If anyone had this problem before or has an idea on how to fix this, it would be very much appreciated.

[mholt]

The ACME server must set the Order state to valid after the call to finalize, but it is leaving it in the ready state, which means "ready to finalize", but we already finalized it. So that is an error. Other ACME clients might be more lenient but ACMEz (which CertMagic uses) requires proper spec compliance in this regard.

You can see the code here where the error comes from:

https://github.com/mholt/acmez/blob/396e7d9b0c0fdef24eb2f9f9b38937b751a966db/acme/order.go#L212-L243

It is called from here:

https://github.com/mholt/acmez/blob/396e7d9b0c0fdef24eb2f9f9b38937b751a966db/acme/order.go#L165

If you look up above that at L139, we have already finalized the order and did not get an error doing so, thus we expect valid state.

I could be wrong in my interpretation of the spec, but your CA is the only one I've seen this error from and it'd be useful to be in contact with the developers of the software if that's the case.

[ffly90]

@mholt: thank you for your reply. I will try to get in contact with the operators of the ACME CA, to find out which software stack they are running. When I here back from them I'm happy to give you more insights on the setup.

[ffly90]

@mholt: so far I found out, that the CA Server software is keyon true-Xtender. When I am getting more Information like the installed version, I will let you know.

[francislavoie]

Pinging @Thomas-Stu who I've seen comment on behalf of Keyon in https://github.com/win-acme/win-acme/issues/1718. Is this something we can put on your radar?

[ffly90]

After some radio silence I got an interesting information from the operations team that are responsible for the keyon CA:

I analyzed the error log file of ACME and found the failure reason “The pkcs10 validation failed due to: The actual key size 256 does not meet the minimum keysize 2048”

I suspect that the problem is, that certmagic uses ed25519 keys by default, whereas the ACME server expects rsa keys. I think it would be very helpful, if there is a way to make certmagic (and caddy for that matter) printing out an error message indicating this. Maybe changes on the ACME server are needed.

[ffly90]

Additionally the keyon support provided this information:

We assume that true-xtender RA and the associated template are currently configured exclusively for RSA. In this case, the ECC request would be rejected by the CA because the associated template is for RSA.

The ACME client "caddy / certmagic" should also be able to generate Pkcs10 requests with RSA keys so that you can test ACME successfully. The client should have saved the Pkcs10 request locally as a file. For example, you could use "certutil -dump [Pkcs10File]" to check which algorithm was used to generate the key pair.

If you want to have both options (signing with RSA or with ECC), you would have to set up two different Microsoft templates at the CA. (which can then have different minimum key lengths) and also configure them in the true-xtender RegistrationAuthority, as a separate CertificateProfile. For this you would have to approach the true-xtender RA responsible person at your place. In this case there would also be two different ACME Urls that the clients then have to use (namely one for RSA and one for ECC certificates).

[francislavoie]

Certmagic's default key type is ECC prime256:

https://github.com/caddyserver/certmagic/blob/b6b3db32bcd5f87d56bc0fba1db708499259653a/crypto.go#L348

Let's Encrypt and ZeroSSL don't support ed25519 yet, so that's the best default to use right now.

You can adjust Certmagic's configuration to use rsa2048 if that's what your CA needs.

https://github.com/caddyserver/certmagic/blob/924f3153e93ec666de747fd3367610f929aba580/config.go#L80-L82

Just set that field to this:

    certmagic.Default.KeySource = certmagic.StandardKeyGenerator{
        KeyType: certmagic.RSA2048,
    }

[ffly90]

@francislavoie thank you! I was just going through the code and also realized that p256 is the default. The issue stays the same. Thank you for providing the correct configuration to use RSA2048!