lbelkind commented 7 years ago

In January 2018 Lets Encrypt will launch a new ACME v2 Endpoint that will allow wildcard certificates. Any plans to implement this?

GeorgeRT commented 6 years ago

Is your mic on?

lukas2511 commented 6 years ago

Yes! I have a few use cases for wildcard certificates too, so this will definitively get implemented!

lbelkind commented 6 years ago

Great news! Do you have any timeline? Is there any way we can collaborate / help you in this effort?

Leonid Belkind, Co-Founder & CTO, Luminate Security Mobile: +972-52-3476783 leonid@luminatesec.commailto:leonid@luminatesec.com  [id:image001.png@01D3677E.939CE6B0]

This message may contain confidential and/or proprietary information, and is intended only for the person / entity to whom it was originally addressed.The content of this message may contain private views and opinions which do not constitute a formal disclosure or commitment unless specifically stated.

From: Lukas Schauer notifications@github.com Reply-To: lukas2511/dehydrated reply@reply.github.com Date: Thursday, 14 December 2017 at 1:30 To: lukas2511/dehydrated dehydrated@noreply.github.com Cc: Leonid Belkind leonid@luminatesec.com, Author author@noreply.github.com Subject: Re: [lukas2511/dehydrated] ACME v2 Support (#420)

Yes! I have a few use cases for wildcard certificates too, so this will definitively get implemented!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/lukas2511/dehydrated/issues/420#issuecomment-351558980, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AXVzSfParsPJJ9oBTpPq0yXIfcqGPj-dks5tAF4igaJpZM4OyNHQ.

lukas2511 commented 6 years ago

I haven't actually read the spec yet... If not too much has changed this may be implemented in January, certainly only pushed to master after the new endpoint is online. If it's more time consuming I'm guessing late Feb or Mar. Certificate handling probably doesn't need to be touched a lot, support for wildcard certificates should be easily handled (script-internally) as if they would just be a normal subdomain, I'm kinda hoping that only a few api calls have to be changed.

A great help would be a short summary (if in any way possible...) of the major changes in the protocol.

If a lot has changed a proof of concept shell or python (or some other easily readable language) script with api calls against the new version would be great. Just something that gives a good starting point for figuring out how stuff works, nothing more.

lbelkind commented 6 years ago

Can’t say that I am familiar with the current Boulder implementation of ACME enough to comment on the diff. I am looking at the latest IETF-standardized ACME implementation (which, according to the ITRG people will serve as the basis for their ACME V2 Endpoint): https://tools.ietf.org/html/draft-ietf-acme-acme-07 [this is, at least, the latest available draft] I am looking at the RFC diff: https://tools.ietf.org/rfcdiff?url1=https://tools.ietf.org/id/draft-ietf-acme-acme-01.txt&url2=https://tools.ietf.org/id/draft-ietf-acme-acme-07.txt The protocol was enriched significantly, the question is, what is the minimal delta to support the new capabilities (My personal interest in ACME V2 is around the wildcard certificates).

From: Lukas Schauer [mailto:notifications@github.com] Sent: Friday, December 15, 2017 3:49 AM To: lukas2511/dehydrated dehydrated@noreply.github.com Cc: Leonid Belkind leonid@luminatesec.com; Author author@noreply.github.com Subject: Re: [lukas2511/dehydrated] ACME v2 Support (#420)

I haven't actually read the spec yet... If not too much has changed this may be implemented in January, certainly only pushed to master after the new endpoint is online. If it's more time consuming I'm guessing late Feb or Mar. Certificate handling probably doesn't need to be touched a lot, support for wildcard certificates should be easily handled (script-internally) as if they would just be a normal subdomain, I'm kinda hoping that only a few api calls have to be changed.

A great help would be a short summary (if in any way possible...) of the major changes in the protocol.

If a lot has changed a proof of concept shell or python (or some other easily readable language) script with api calls against the new version would be great. Just something that gives a good starting point for figuring out how stuff works, nothing more.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/lukas2511/dehydrated/issues/420#issuecomment-351890016, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AXVzSWWmB3qd2rzABmIIg_rGd_5K_boiks5tAc_zgaJpZM4OyNHQ.

cpu commented 6 years ago

I am looking at the latest IETF-standardized ACME implementation (which, according to the ITRG people will serve as the basis for their ACME V2 Endpoint): https://tools.ietf.org/html/draft-ietf-acme-acme-07 [this is, at least, the latest available draft]

@lbelkind That's actually two drafts behind the latest :-) Draft-09 is the latest and was cut a few days ago. The Boulder divergences doc is a little bit behind.

A great help would be a short summary (if in any way possible...) of the major changes in the protocol.

@lukas2511 I'd love to eventually sit down and write something like this but I'm still tied up implementing the new protocol & getting ready for launch in Jan/Feb.

Here's a (off-the-top-of-my-head) summary of some major changes that will hopefully help:

The authorization/issuance flow has changed. Prev. you requested your own authz's per-identifier and then submitted a CSR to new-cert. Now you submit a request for an order for a set of identifiers, the server gives you back authorizations to satisfy, and you issue a certificate by POSTing a CSR to the order's finalize URL.
The JWS authorization of requests has changed. Only the new-account JWS has an embedded JWK now, all other endpoints consume a JWS with a key ID.
JWS bodies don't include the "resource" field anymore, the client needs to put the URL it intends the message for in a protected header of the JWS.
Lots of resources changed to be camelCase throughout (revokeCertificate, newAccount, etc)

In terms of LE specific things to know:

Order identifiers with a wildcard prefix will be given an authorization with only a DNS-01 challenge to solve.
The TLS-SNI-01 challenge type won't be present in the V2 API - We'll only offer its successor TLS-SNI-02

Those are some of the bigger changes that come to mind right off the bat. Definitely not a comprehensive list. I would certainly welcome a community contribution of a more thorough document as a PR to the Let's Encrypt website if someone is feeling like spending more time diffing RFCs :-)

lukas2511 commented 6 years ago

@cpu thank you. that doesn't sound too bad, sure a lot of changed things but it actually sounds quite easy to implement and a lot faster and less error-prone than the previous api.

ginkel commented 6 years ago

FYI: The staging v2 endpoint is now available: https://community.letsencrypt.org/t/staging-endpoint-for-acme-v2/49605

rudiedirkx commented 6 years ago

Only DNS-01 validation for wildcard domains =( That screws with my full automation. Will Dehydrated support plugins for adding DNS APIs?, since every domain provider has its own API.

jamgregory commented 6 years ago

@rudiedirkx - Dehydrated already supports DNS hooks (I'm currently using the one for Cloudflare, which works great - https://github.com/kappataumu/letsencrypt-cloudflare-hook). Not sure whether you know Python at all, but you might be able to modify that to your own purposes?

rudiedirkx commented 6 years ago

Wait waat! I did not know that. I will reread Dehydrated. Python should work, or any other language if it's just a bash call. Thanks all.

txr13 commented 6 years ago

@rudiedirkx You should look into https://github.com/AnalogJ/lexicon as that allows you to use a standardized syntax to interface with multiple DNS providers.

belliash commented 6 years ago

Any update soon planned?

crza commented 6 years ago

464 contains support for v2 API.

Regex handling seems to block wildcard domains, but I don't have time to look at that.

belliash commented 6 years ago

So its still useless

txr13 commented 6 years ago

@belliash That seems harsh, considering how much of the v2 API has been implemented.

MDXDave commented 6 years ago

@belliash production endpoint v2 will only be available at 27th february anyway.

lukas2511 commented 6 years ago

I have merged @crza's initial implementation. There still seems to be a bit of work required and I didn't yet test if it actually works, but if anybody wants to give it a try, feel free: Replace the CA variable and set API=2 in the config.

lukas2511 commented 6 years ago

I was able to test the changes and I was able to acquire a few certificates (including wildcard certificates :tada:) from the staging API.

There are still a few issues, e.g. I had an invalid authorization that I just wasn't able to get rid of, I didn't have much time to look into that issue, it seemed like a server-side bug to me, but I'll have to look into this a bit more.

cpu commented 6 years ago

I had an invalid authorization that I just wasn't able to get rid of, I didn't have much time to look into that issue, it seemed like a server-side bug to me, but I'll have to look into this a bit more.

@lukas2511 this is https://github.com/letsencrypt/boulder/issues/3333 and indeed a server-side bug.

lukas2511 commented 6 years ago

@cpu thanks for the clarification! definitively saved me from a few more hours of debugging this issue 😃

chriscpritchard commented 6 years ago

Worked for me on the staging endpoint, was able to request certs for various subdomains and a wildcard cert.

rudiedirkx commented 6 years ago

I'm getting several errors while testing a SAN certificate.

The Dehydrated feedback has changed (maybe because the ACME flow has changed).

Old:

 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for cn.example.com...
 + Already validated!
 + Requesting challenge for san1.example.com...
 + Requesting challenge for san2.example.com...
 + Requesting challenge for san3.example.com...
  <dns hook output, with HOOK_CHAIN>
 + Responding to challenge for san1.example.com...
 + Challenge is valid!
 + Responding to challenge for san2.example.com...
 + Challenge is valid!
 + Responding to challenge for san3.example.com...
 + Challenge is valid!
  <dns hook output, with HOOK_CHAIN>
 + Requesting certificate...

Very obvious that 1 / 4 domains was already validated, and 3 others are being challenged now.

New:

 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenges for cn.example.com san1.example.com san2.example.com san3.example.com...
 + Already validated!
  <dns hook output, with HOOK_CHAIN>
 + Responding to challenge for cn.example.com...
  + ERROR: An error occurred while sending post-request to https://acme-staging-v02.api.letsencrypt.org/acme/challenge/C1FKeAYUL2svnxV8eaGd7QUWxv3GLxPD-ElxqiMnBlY/96144727 (Status 400)

Details:
{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Unable to update challenge :: The challenge is not pending.",
  "status": 400
}

The strange error aside, this feedback is much less clear. Which domains were valid? Which are passed to the dns hook?

And then the strange error of course: challenge not pending..? It tries 1 of the 4, but not the first, but that shouldn't matter.

I'm requesting a new version of an existing certificate (new SAN domains, same CN domain). I think v2 creates a new account automatically? I had to --register --accept-terms anyway.

rudiedirkx commented 6 years ago

For a new certificate with a new account, it all works, but that's probably not the usual case after having used Dehydrated before.

lukas2511 commented 6 years ago

@rudiedirkx the "challenge not pending" bug is what @cpu was talking about, this is a serverside bug and not a dehydrated bug, the server returns an "invalid" challenge that can't be cleared and dehydrated doesn't expect that to happen as it should never happen.

other than that what exactly is your problem with the output not being clear? it clearly states what domain it's checking. also there are differences on how challenges are generated, they no longer have to be requested domain by domain, they will be delivered by the server as part of the "order", and dehydrated will only tell the server that the challenges are ready (which still happens for each domain)

rudiedirkx commented 6 years ago

 + Requesting challenges for cn.example.com san1.example.com san2.example.com san3.example.com...
 + Already validated!

That's not very clear to me. Is 1 domain already validated? Or all 4? The old method is unambiguous.

The order is also strange:

 + Responding to challenge for A...
 + Challenge is valid!
 + Responding to challenge for *.A...
  <dns 'clean_challenge' hook output>
  <dns 'clean_challenge' hook output>
  <dns 'clean_challenge' hook output>
  <dns 'clean_challenge' hook output>
  <dns 'clean_challenge' hook output>
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "dns-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "Incorrect TXT record \"odpkYcpEaObZt06djoRGBD3VVFbkF3qCyo89bJ15X7E\" found at _acme-challenge.A",
    "status": 403
  },
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/r7Z8k24C7Hk0lF-QMVUTHCX2ty6qe1427mUdKO1OcYI/96153797",
  "token": "vwGI10QVYDV82eoBXO0q_AJ0ovgl64TrbGnwPhj5FMM",
  "keyAuthorization": "vwGI10QVYDV82eoBXO0q_AJ0ovgl64TrbGnwPhj5FMM.xVneMCNmuRpkmgiO43PGeoDwTtuaqYlo8URll2rKRJI"
})

The DNS hook is called in between validations? Or after the first failure, but before printing that failure?

(Should the challenge for *.foo.example.com be in _acme-challenge.foo.example.com or _acme-challenge.example.com? I'm getting mixed reports, and mixed results...)

Maybe it's all me. This is my first time acmev2ing.

rudiedirkx commented 6 years ago

It's getting stranger and stranger:

 + Requesting challenges for wc13.example.com *.wc13.example.com *.example.com...
  <dns hook output>
 + Responding to challenge for wc13.example.com...
 + Challenge is valid!
 + Responding to challenge for *.wc13.example.com...
 + Challenge is valid!
 + Responding to challenge for *.example.com...
  <dns hook output>
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "dns-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "Incorrect TXT record \"putPxdJTu37e0FDnXekp3e5kYR3jAUdSqrIz4KO6kVk\" (and 1 more) found at _acme-challenge.wc13.example.com",
    "status": 403
  },
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/_5AY7BT_nCtXgaub0dJhUQVFON4v-kYHusi41-ZbWxg/96160910",
  "token": "rTMVk2UGtrzlghg2LaiHzYHnnTU5bhfaT1uMMCZrDDU",
  "keyAuthorization": "rTMVk2UGtrzlghg2LaiHzYHnnTU5bhfaT1uMMCZrDDU.xVneMCNmuRpkmgiO43PGeoDwTtuaqYlo8URll2rKRJI"
})

It's Responding to challenge for *.example.com... but finds the wrong record at _acme-challenge.wc13.example.com. Why would it be looking at _acme-challenge.wc13.example.com? There must be something wrong with the feedback order.

Sometimes all validations succeed and I get a cert, and sometimes not... V1 mode always works always, but no wildcards of course.

rudiedirkx commented 6 years ago

There's definitely something wrong, and it's not just the feedback display.

Processing wc14.example.com with alternative names: *.wc14.example.com *.example.com
 + Signing domains...
 + Creating new directory /var/www/letsencrypt/le-config/home-staging/certs/wc14.example.com ...
 + Generating private key...
 + Generating signing request...
 + Requesting challenges for wc14.example.com *.wc14.example.com *.example.com...
 + Already validated!
  <dns hook 'deploy_challenge' output for '*.wc14.example.com' and '*.example.com'>
 + Responding to challenge for *.wc14.example.com...
 + Challenge is valid!
 + Responding to challenge for *.example.com...
  <dns hook 'clean_challenge' output>
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "dns-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "Incorrect TXT record \"8dJ02P5CRBVmEoW5stSyLSV-MbQPIRkivYf5XFkS8LQ\" found at _acme-challenge.wc14.example.com",
    "status": 403
  },
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/-DAlRTcc0kKvu249mV5shlH_SpZqfYQF-sAc90Bw4Rg/96163527",
  "token": "ZIVZsdXSmROyaVC_RiNZbvKAzNa38c8P_EjHtxsqRUY",
  "keyAuthorization": "ZIVZsdXSmROyaVC_RiNZbvKAzNa38c8P_EjHtxsqRUY.xVneMCNmuRpkmgiO43PGeoDwTtuaqYlo8URll2rKRJI"
})

I've definitely never ever validated wc14.example.com before, but example.com might be valid. Dehydrated must think wc14.example.com is valid, because it fires the dns hook for *.wc14.example.com and *.example.com.

So something in the new V2 flow is wrong. And now I'll stop spamming.

lukas2511 commented 6 years ago

@rudiedirkx there was a missing log statement, i added it in 5fd93ea87422c6e47f71c2ae8055303cb6e410ff, it should now be clear what the script is doing.

Your last issue seems to be that both *.wc14.example.com and wc14.example.com would have to be validated at the same record, this can't be supported with HOOK_CHAIN enabled as both records would have been set at the exact same time.

I'll add a warning and an automatic fallback to non-chaining behaviour for this. That should solve the issue. With wildcards you shouldn't have too many domains in that certificate, so that shouldn't be much of a problem anyway.

lukas2511 commented 6 years ago

Additionally to the hook-chaining problem I found another flaw that might lead to swapped challenges, which in turn will result in invalid authorizations.

I'll be rewriting most of the authorization-handling-code as in it's current form it's basically broken with ACME v2.

Thanks for bringing your issues up @rudiedirkx, wouldn't have found that flaw until much later...

rudiedirkx commented 6 years ago

I think ACME and LE support multiple records on the same domain to validatie different domains. A very normal scenario is *.example.com and example.com. ACME will require 2 TXT records in _acme-challenge.example.com. It seems (from the many error messages I've read) LE looks at all the TXT records.

When you've fixed the challenge swapping problem, I'll try my rubbish again. I really hope HOOK_CHAIN still works in all v2 scenario's, because it's a huge speed gain.

rudiedirkx commented 6 years ago

Also, maybe a different issue, wildcard dirs:

Creating new directory certs/*.example.com ...

Not every OS/fs loves dirs starting with (or even containing) a *. Maybe replace with - or __ or something else less-dangerous-but-still-unique?

rudiedirkx commented 6 years ago

Also, for a cert with only 1 domain, a wildcard, already validated:

 + Found valid authorization for example.com
 + 0 pending challenge(s)
../../letsencrypt.sh/dehydrated: line 710: challenge_identifier[*]: unbound variable
 + Requesting certificate...
 + Checking certificate...
 + Done!

It seems to have worked anyway though...

rudiedirkx commented 6 years ago

All my weird challenge validation tests work with your changes, and the feedback is great. Thanks!

lukas2511 commented 6 years ago

@rudiedirkx Oh, boulder can validate with multiple TXT records set? I didn't know that, that changes a lot and would also resolve the issue I filed with boulder (https://github.com/letsencrypt/boulder/issues/3405). I'll look into the undefined variable thing and will test the hook chaining thing, if it works I'll remove my "workaround".

lukas2511 commented 6 years ago

@rudiedirkx You were right, deploying multiple tokens works perfectly! I'm going to change the loop behaviour a bit so it will work with both HOOK_CHAIN enabled and disabled.

lukas2511 commented 6 years ago

Mentioned changes were implemented in afba7c694cb0a6d8f150e2c5e2bd86a3c688312a.

rudiedirkx commented 6 years ago

Seems to work very nicely! Very good feedback. No error if 0 pending challenges. With any domain order. With and without wildcards. Only tried with HOOK_CHAIN enabled because duh.

I can't wait for Feb 27. I'm guessing the LE servers will be very busy.

belliash commented 6 years ago

Requesting new certificate order from CA...
Received 1 authorizations URLs from the CA
Handling authorization for test.org
1 pending challenge(s)
Deploying challenge tokens... Waiting 600 seconds to propagate DNS changes
Responding to challenge for test.org authorization...
Challenge is valid!
Requesting certificate...
- ERROR: An error occurred while sending post-request to https://acme-staging-v02.api.letsencrypt.org/acme/order/5358218/3754/finalize-order (Status 403)

Details: { "type": "urn:ietf:params:acme:error:unauthorized", "detail": "Error finalizing order :: Order includes different number of names than CSR specifies", "status": 403 }

CSR included test.org and *.test.org So something is not working with CSR, when launching dehydrated --accept-terms --ocsp --signcsr ./test.csr

lukas2511 commented 6 years ago

@belliash should be fixed with 45f5c17260a7a76a392f11c8a4c762c355476ba0

belliash commented 6 years ago

INFO: Using main config file /etc/dehydrated/config

Requesting new certificate order from CA...
Received 2 authorizations URLs from the CA
Handling authorization for test.org
Handling authorization for test.org
2 pending challenge(s)
Deploying challenge tokens...

array(2) { ["file"]=> string(43) "qrJB4iHDj31jBnoRob5Cupm0h-xmtoGNqC3vYJOy220" ["token"]=> string(43) "A6jqehG6Tei6JkYo6-4S4xIjXzPnP-3N4XCzzzUfatk" } array(2) { ["file"]=> string(43) "8GvVlpAkr_vESJJNYGDTwXS0oW6LrlwhNsDanZajmJA" ["token"]=> string(43) "qNipbS_bokld5wp3IYyGn36xixuWNE2MGV6_aIduCCI" } Waiting 600 seconds to propagate DNS changes

Responding to challenge for test.org authorization...
- ERROR: An error occurred while sending post-request to https://acme-staging-v02.api.letsencrypt.org/acme/challenge/28wW8W6Y01GR7EbeqERTKKnKgZcnzzRbmU6sUv6ml_U/97842001 (Status 400)

Details: { "type": "urn:ietf:params:acme:error:malformed", "detail": "Unable to update challenge :: The challenge is not pending.", "status": 400 }

As you can see the autorization is for test.org twice... not test.org and *.test.org. Im not usre if its ok or not, but afterwars, my DNS hook script written in PHP has added the following entries:

_acme-challenge.test.org. 60 IN TXT "A6jqehG6Tei6JkYo6-4S4xIjXzPnP-3N4XCzzzUfatk" _acme-challenge.test.org. 60 IN TXT "qNipbS_bokld5wp3IYyGn36xixuWNE2MGV6_aIduCCI"

so it looks ok to me, but anyway I cannot obtain wildcard certificate. Any ideas what is wrong?

lukas2511 commented 6 years ago

@belliash nothing wrong, that's exactly how it's supposed to be, wildcards are validated at the upper level domain (and there is no way to differentiate which authorization is for which domain, so I can't make it more verbose). the issue you are running into seems to be the boulder bug that was already mentioned in this discussion: https://github.com/letsencrypt/boulder/issues/3333. it's fixed now, but I guess it isn't yet deployed to the staging environment. as a workaround you can delete your account key and register a new one, that should work.

lukas2511 commented 6 years ago

I'm closing this as ACME v2 support seems to be quite stable by now 🎉

There are still a few things that might be implemented before the next release (e.g. order-reuse after certain failures would be nice), but main support for the new protocol and wildcard certificates is there and seems to be working fine 😄

If you are having issues with ACME v2 please create new issues instead of replying to this one.

bviktor commented 6 years ago

+1 for *.domain.com not being a good idea. Most CAs usually use star_domain_com, we might as well resort to star_domain.com.

bviktor commented 6 years ago

How do I tell dehydrated to request a wildcard cert? Can't seem to find this info anywhere. Is it *.domain.com in domains.txt?

lukas2511 commented 6 years ago

@bviktor *.domain.com in domains.txt, just like every other domain. but keep in mind that it only works with ACMEv2, so currently only with the staging api.

bviktor commented 6 years ago

Yeah, thanks, successfully requested a wildcard one from staging v2, no change needed for the DNS hook, sweet :) Can't wait for prod rollout. Will there be no dns-02? Coz I assumed so.

rudiedirkx commented 6 years ago

v2 should be available, no? https://letsencrypt.org/2017/06/14/acme-v2-api.html But where's the directory? And the announcement?

jobe1986 commented 6 years ago

LetsEncrypt has delayed the release of the production ACME v2 API end point. See https://letsencrypt.org/2017/07/06/wildcard-certificates-coming-jan-2018.html and the linked post at https://community.letsencrypt.org/t/acmev2-and-wildcard-launch-delay/53654 for more information.

nogweii commented 6 years ago

And now the ACMEv2 API is public with wildcard support! 🎉 https://community.letsencrypt.org/t/acme-v2-and-wildcard-certificate-support-is-live/55579

Congrats on the hard work, LE team! Thanks @lukas2511 for being ahead of this and landing ACMEv2 support already!

dehydrated-io / dehydrated

ACME v2 Support #420

464 contains support for v2 API.