golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.37k stars 17.58k forks source link

doc: add ACME (LetsEncrypt, etc) example docs to the standard library #17053

Open bradfitz opened 8 years ago

bradfitz commented 8 years ago

Thanks to @crhym3, we now have a suitably-licensed & CLA-clean ACME implementation in https://godoc.org/golang.org/x/crypto/acme (and a high-level package in https://godoc.org/golang.org/x/crypto/acme/autocert).

I'd like to consider privately vendoring that into Go 1.8 and making HTTPS even easier.

I'd like a complete user program with automatic HTTPS certs to look something like:

package main

import (
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", websiteHandler)
    srv := &http.Server{
        Addr:         "example.com:443",
        ACMEServer:   http.LetsEncrypt, // non-empty enables autocert support
        ACMEAgreeTOS: func(tosURL string) bool { return true },
        ACMEEmail:    "foo@bar.com", // (but optional)
    }
    log.Fatal(srv.ListenAndServeTLS("", ""))
}

Misc notes:

My goal is for HTTPS to be dead simple out of the box.

Thoughts?

/cc @adg @broady @campoy @quentinmit @rsc @robpike @ianlancetaylor @mholt @crhym3

mholt commented 8 years ago

Like what I'm seeing so far!

I wonder if the ACME configuration should be in a separate struct value -- do we want to tether the http.Server type to ACME concretely?

One of the requests we've had in Caddy is to abstract the way certificates are Obtain()ed and Renew()ed -- in other words, an interface with approximately these two methods. An ACME client would be one implementation, a hashicorp/vault implementation might be another, etc. So I wonder if, instead, there should be some sort of interface type (TLSManager? Not sure what you'd call it), which some type tls.ACMEClient implements (or something like that).

In fact, thinking on these lines, it might be beneficial to drop down to the tls package somehow when working with ACME. In other words, you create a TLS listener that has some notion of ACME, but the HTTP server doesn't care. It's just one application of TLS and uses that listener and goes about its usual business.

FWIW, In Caddy I found myself writing essentially a wrapper type over tls.Config that specifies all the ACME stuff and makes the GetCertificate callback, etc. It all happens within TLS (not just HTTPS), so how about an easy way to make an ACME-capable TLS listener that you pass into http.Serve()?

nhooyr commented 8 years ago

I've been moving all of my Go servers to use ACME (with letsencrypt) and I've been using Russ's letsencrypt package for now because it is so simple and easy. However, I've also contributed to the acme package that @crhym3 wrote because its design was inspired by Russ's package and because I wanted ACME to be in the stdlib. I think it is a very common use case that we should cater towards.

However, I don't like the use of fields in http.Server to configure ACME. It's very easy to just create a tls.Listener using acme.Manager's GetCertificate callback. However for convenience, it just makes more sense for this to be part of crypto/tls rather than purely net/http so that even generic TLS servers can use ACME more easily.

(as I was writing this, I noticed @mholt's comment which I 👍)

rolandshoemaker commented 8 years ago

This is super cool!

With my Let's Encrypt hat on: One important thing to consider though is that currently neither the ACME spec nor the API implementation provided by Let's Encrypt should be considered completely stable. The Let's Encrypt API currently implements a amalgam of the four ACME RFC drafts which we colloquially refer to as v01. Once the RFC is finalized we intend to implement a new API version, v02, which will be a strict implementation of the final specification language which will not be completely backwards compatible with v01.

Once v02 is made public (we are currently aiming for somewhere around the start of 2017) we will provide a timetable for depreciating the v01 implementation . Given this might it make sense to wait until the finalized stable API is available before adding this support to the stdlib in order to prevent churn and breakage in users who don't update?

tmornini commented 8 years ago

My goal is for HTTPS to be dead simple out of the box

👍🏻

bradfitz commented 8 years ago

Once v02 is made public (we are currently aiming for somewhere around the start of 2017) we will provide a timetable for depreciating the v01 implementation . Given this might it make sense to wait until the finalized stable API is available before adding this support to the stdlib in order to prevent churn and breakage in users who don't update?

I totally agree. But Go 1.8 isn't due until 2017 anyway. The timing might work out? But even Go 1.9 is acceptable, if that works out best timing-wse.

jsha commented 8 years ago

Another Let's Encrypt developer here. Wrote out a post saying what @rolandshoemaker said about upcoming ACME changes, but he beat me to it!

Also, ideally the design should encourage disk-based caching as much as possible. If Go programs are designed to issue on each startup, people will very quickly run into Let's Encrypt rate limits. I see that autocert already has a Cache, so it's probably enough to include the Cache mechanism in the library and make sure the examples use it.

Lastly: This is great, and I'm super happy to see it. This is exactly the type of thing Let's Encrypt exists to enable. Thanks!

bradfitz commented 8 years ago

@jsha, as I said in my first post, caching would automatic. There wouldn't even be an option to disable it. Only an option to change the directory it uses.

jsha commented 8 years ago

Got it! I read too quickly. :-)

dlsniper commented 8 years ago

As awesome as this sounds, I think this shouldn't be included in the standard library.

I understand that people will say: it's about security, or other arguments, which, yes, are valid.

But please consider this:

Considering proposals to include different useful things in the language or standard library have been shut down of over the course of years, I would like to know if this means a change in vision on how the standard library / language should evolve from here on out. Does it mean that I can raise a proposal for having ? : syntax in the language? It is super useful in so many cases but it has been deemed not necessary as we can do if / else.

bradfitz commented 8 years ago

@mholt, @nhooyr, as I wrote:

The ACMEFoo names are placeholders. Maybe they'd deserve their own struct.

I agree they're probably too stuttery. Being a listener is interesting, though. Maybe:

package main

import (
    "crypto/tls"
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", websiteHandler)
    var srv http.Server
    log.Fatal(srv.Serve(&tls.ACMEListener{
        Domain:   "foo.com",
        Provider: tls.LetsEncrypt,
        AgreeTOS: tls.YesIAgreeTOS,
    }))
}
nhooyr commented 8 years ago

@bradfitz I like that. But now, I'm not sure if crypto/tls is the best place for such a listener. Perhaps we should add acme.Listener?

bradfitz commented 8 years ago

@dlsniper, the timing with regards to its draft status is discussed above. See comments from @rolandshoemaker and @jsha.

the functionality exists outside of the standard library just fine

Yes. And maybe all we do here is add documentation examples on how to use autocert. Maybe that's enough. I'd like it to be even easier, though.

Does it mean that I can raise a proposal for having ? : syntax in the language?

You can propose anything you like, but that discussion happened a long time ago and was already decided. If you insist on raising it again, please do it elsewhere and not in this issue.

bradfitz commented 8 years ago

@nhooyr, I don't really want to add a new package, though.

We already have https://golang.org/pkg/net/http/httputil/ though. Maybe that's a suitable location.

mholt commented 8 years ago

@bradfitz That's better. I imagine the tls.ACMEListener will listen only on port 443?

And maybe the Provider value could use a sensible default. Let's Encrypt's endpoint is a good one -- the question is, should it be their staging endpoint or production one? If people are testing their Go program against the "default" endpoint which happens to be the production endpoint, especially if they're wiping their file system (containers?) or cleaning a test directory or something, it could hit CA rate limits pretty fast. So maybe, on second thought, they should be required to specify an endpoint either way...

And why not just true for agree TOS?

nhooyr commented 8 years ago

@bradfitz actually, I meant the high level package https://godoc.org/golang.org/x/crypto/acme/autocert.

nhooyr commented 8 years ago

So

package main

import (
    "crypto/acme"
    "crypto/acme/autocert"
    "crypto/tls"
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", websiteHandler)
    var srv http.Server
    log.Fatal(srv.Serve(&autocert.Listener{
        Domain:   "foo.com",
        Provider: acme.LetsEncrypt,
        AgreeTOS: autocert.AcceptTOS,
    }))
}

I think it would fit in with autocert being the high level package for using acme.

bradfitz commented 8 years ago

@mholt, I addressed my opinion on the default in my initial proposal text at top:

ACMEServer would be required as the opt-in, and we wouldn't make LetsEncrypt be automatic or preferred, but we would add a constant const LetsEncrypt = "https://acme-v01.api.letsencrypt.org/directory" like the acme package has.

We could provide two LetsEncrypt constants for prod vs staging but I don't want to make LetsEncrypt be automatic. Depending on any specific organization (for-profit or otherwise) rubs me the wrong way. I want users to explicitly opt-in to it and know what's happening. (Or hopefully know what's happening by being curious what the line Provider: acme.LetsEncrypt means)

nhooyr commented 8 years ago

@mholt the acme package uses a callback to determine whether or not you agree to the TOS. the autocert package provides a convenient AcceptTOS function that always returns true. Maybe we could just remove it? I think pretty much everyone just agrees, even if they don't read it. For people who actually care, they can use autocert.

mholt commented 8 years ago

@nhooyr

I think it would fit in with autocert being the high level package for using acme.

The only issue there is that you then have to import an external package to use it.

bradfitz commented 8 years ago

@nhooyr, you can basically already do that. This proposal is about whether Go should come out of the box with ACME support, now that ACME is a thing. Go has always been very pro-easy-HTTPS, but ACME wasn't an option at the time when Go first came out.

The question is whether we continue the tradition of promoting easy HTTPS.

bradfitz commented 8 years ago

@nhooyr,

Maybe we could just remove it? I think pretty much everyone just agrees, even if they don't read it.

I already addressed that. See my original text at the top of this issue.

mholt commented 8 years ago

@bradfitz

I addressed my opinion on the default in my initial proposal text at top

Oops, you're right.

Depending on any specific organization (for-profit or otherwise) rubs me the wrong way. ... I want users to explicitly opt-in to it and know what's happening.

This makes sense. And I agree with this philosophy. But if you're going to provide a const for Let's Encrypt, would you have to do that for all public ACME CAs? Maybe the user should just provide their own CA URL string.

nhooyr commented 8 years ago

@bradfitz I understand now. In that case, the current approach with adding it to crypto/tls seems fine to me (though I'd definitely prefer a separate package but I can see why you don't want add more packages to the stdlib). I was under the impression that the acme package would be added to the standard library which would make it's inclusion in crypto/tls or net/http redundant. I think I remember @crhym3 mentioning it somewhere. I'll try and find it. (edit: never mind I misinterpreted, sorry).

@mholt I thought Lets Encrypt was the de facto ACME CA? I think it's used enough to warrant it's inclusion as a package level const because I am not even aware of any other public ACME CA.

bradfitz commented 8 years ago

But if you're going to provide a const for Let's Encrypt, would you have to do that for all public ACME CAs?

Sure. Or for all public, non-profit (or at least free) ones. I'd rather the user not have to provide a big ugly string. I'd like to keep the boilerplate as tidy as possible.

bradfitz commented 8 years ago

@nhooyr,

I thought Lets Encrypt was the de facto ACME CA? I think it's used enough to warrant it's inclusion as a package level const because I am not even aware of any other public ACME CA.

Let's Encrypt created the ACME spec, but anybody can implement it. StartCom/StartSSL has announced they plan to use ACME: https://www.ietf.org/mail-archive/web/acme/current/msg01290.html So maybe there will be two. We'll see.

nhooyr commented 8 years ago

@bradfitz Why would we need a second public free CA for ACME? StartCom/StartSSL wants to use it for customers which according to your earlier comment means it wouldn't be included and I agree. For anyone starting another public free ACME CA this seems relevant.

robpike commented 8 years ago

The proposal would dramatically increase the amount of code net/http depends on. For that reason alone, I'd prefer to leave ACME support where it is and settle on clean, standard way to use it. I do not believe it is necessary for it to be in the standard repo for it to be easy to use.

To put it another way, you have stated your goal of easy security, which I applaud, but there has not been sufficient study of the various ways to do that.

bradfitz commented 8 years ago

@robpike, it's not much code, if that's the root of your argument.

bradfitz commented 8 years ago

To put it another way, you have stated your goal of easy security, which I applaud, but there has not been sufficient study of the various ways to do that.

I don't even know what that means. I don't think you mean that the Go project investigate sending all users 2FA hardware devices, for instance. This bug is about HTTPS and certs.

The modern web is moving towards HTTPS. All new browser features are requiring HTTPS. I want to make using HTTPS and getting today's browser features as easy as it was to run a website and get the latest browser features in 2012.

nhooyr commented 8 years ago

@robpike Plus, it would be great for PR. If this sort of change is merged, I'm certain it will be received well by the community (see +1s on this proposal). People want the web to be more secure, and if they see Go embrace lets encrypt in its stdlib, that's another reason to use Go. Even if it is easy to just import the autocert package and create the manager yourself, just the fact that it is in the stdlib would mean something to people.

And it increases visibility by being in the standard library docs which also means more people would use it.

rasky commented 8 years ago

As soon as it interacts cleanly and easily with net/http, I don't see "being in the standard library" as adding much convenience. You're basically saving one import line to users. IOW, given the code in https://github.com/golang/go/issues/17053#issuecomment-246138060, I don't think we can say that this is a lot less convenient:

package main

import (
    "crypto/tls"
    "log"
    "net/http"
    "golang.org/x/crypto/acme"
)

func main() {
    http.HandleFunc("/", websiteHandler)
    var srv http.Server
    log.Fatal(srv.Serve(&acme.Listener{
        Domain:   "foo.com",
        Provider: acme.LetsEncrypt,
        AgreeTOS: acme.YesIAgreeTOS,
    }))
}

At the end of the day, must users would be copy&pasting it anyway and start building upon it.

I'm not against inclusion (though ACME is still in draft, as discussed). But I can't see inclusion as providing such a big value, given this smooth integration.

kjk commented 8 years ago

@rasky being in standard library and documented as part of it is huge convenience.

I was just researching this yesterday and it took me several hours of googling, reading tutorials etc. I found 3 different packages (https://github.com/dkumor/acmewrapper https://godoc.org/golang.org/x/crypto/acme/autocert, https://github.com/rsc/letsencrypt), multiple articles describing (different) setups.

I have no idea which library is the best etc. It's a mess.

From a point of view who's aware of Let's Encrypt and wants to use it, figuring out how to do it in Go is currently a huge time sink.

Having it in standard library is even more important for people who would benefit from having easy https setup but are not even aware of Let's Encrypt and I believe there are many people like that. Being in standard library would have positive effect of advertising Let's Encrypt to more people.

nhooyr commented 8 years ago

@kjk it's pretty clear that it is the Go way if it's in golang.org/x/crypto/. Regardless, I agree that it increases visibility and adding it to the stdlib would make it even more clear. Especially for someone new to the language. I'd wager that the stdlib documentation is read much more by many more people than the golang.org/x/crypto packages.

joneskoo commented 8 years ago

Regardless of what import path is used for ACME support, I think a blog post in the now-very-quiet golang.org blog, announcing an official or semi-official ACME library would be a good idea to raise awareness. This can be done already now. The problem is basically discovering the easy-to-use and high quality, maintained implementation.

Applications need to specifically add support for ACME, and I don't think there's anything proposed here that would change that. Which path is used for import is not important, as long as applications written in Go remove the pain point of acquiring certificates.

I'm not yet convinced that adding of the code to standard library makes a meaningful difference to end-user adoption. However an interesting compromise could be to add an EXAMPLE code to standard library docs, even if the package is external or vendored to Go distribution under x/.

Merovius commented 8 years ago

At some point the official communication was discouraging using go to terminate your TLS connections as go's tls implementation wasn't well-vetted and instead use a reverse proxy or link in OpenSSL. Are we now straying from that recommendation? In a similar vein, I'd consider it more or less bad practice to read/write to disk in a frontend binary, as it limits scalability. So, instead of making it possible to change the directory of the on-disk cache, I'd prefer overwriting the caching-mechanism altogether (still defaulting to an automatic on-disk cache, if you want).

Personally, I think acme lives in x/crypto quite fine. I don't really see a reason to increase the API surface of the stdlib for this. If anything, I'd at least wait O(years) to see how the protocol and it's adoption develops.

x1ddos commented 8 years ago

Moving x/crypto/acme into stdlib also means one doesn't need to vendor it anymore. This is an advantage for simple uses, which is the ultimate goal of this proposal - make it dead simple out of the box.

My only concern is incompatibility with future versions of ACME spec and Let's Encrypt implementation. However, if it becomes stable next year, as already discussed, this shouldn't be an issue in 2017.

I like ACMEListener. A very similar thing was also proposed long time ago in https://github.com/google/acme/issues/14#issuecomment-168226991 by @bradrydzewski.

I'd consider it more or less bad practice to read/write to disk in a frontend binary

@Merovius It's a cache. Certs are served from memory once fetched from cache.

kennylevinsen commented 8 years ago

If saving a single vendor or go get is the only benefit, then I don't think it's worth the effort. As @raski mentions, any improvement in usability does not come from the inclusion in the standard library. The primary benefit of having it in the standard library, is that net/http or other standard libraries could use it directly. That, however, seems entirely unnecessary given the acme.Listener approach, which would provide similar convenience as direct integration.

Having the library be part of the stdlib wouldn't implicitly make it more visible, either. Without an announcement (which could just as well be about golang.org/x), you wouldn't know it was there. Any effort to increase awareness of this package could just as well go towards increasing awareness of a package in golang.org/x.

I would think that making x/crypto/acme even simpler to use (acme.Listen(network, address string, config *acme.Config) (net.Listener, error) or something) would solve the usability problem sufficiently.

harshavardhana commented 8 years ago

Amazing! @bradfitz - thank you.

adg commented 8 years ago

My goal is for HTTPS to be dead simple out of the box.

While this is a laudable goal, I think we should instead make dependency management dead simple. I'm a weak -1 on this proposal, at least until LetsEncrypt have locked everything down.

bradfitz commented 8 years ago

I think we should instead make dependency management dead simple.

True. Maybe that'll happen by the time the ACME protocol is stable. If so, this proposal can simply be about adding some docs/examples to the net/http package about how to do LetsEncrypt in a few lines. I'd be happy with that.

hlandau commented 8 years ago

Some points:

mholt commented 8 years ago

@hlandau

My understanding is that net/http is not 'frontend-hardened'; that it is desirable in production use to put another webserver, such as nginx, in front of it.

What do you mean by this? nginx isn't any more "frontend-hardened" than Go is, unless you configure it a bunch. Go is the same way, but it is certainly production-ready.

Even if an ACME implementation were to be placed in the standard library, this shouldn't be done until the ACME protocol is finalized.

This has already been understood above, as the earliest this would land, if at all, is next year, presumably after ACME is finalized.

@Merovius

At some point the official communication was discouraging using go to terminate your TLS connections as go's tls implementation wasn't well-vetted and instead use a reverse proxy or link in OpenSSL. Are we now straying from that recommendation?

What official communication? What defines well-vetted? Go's crypto/tls package has had much fewer vulnerabilities than OpenSSL has.

In a similar vein, I'd consider it more or less bad practice to read/write to disk in a frontend binary, as it limits scalability.

This behavior can be swapped out. Merely the default is a file system cache, which is fine -- what else are you going to use?

minux commented 8 years ago

@bradfitz, I also remember @agl discouraged using Go TLS to terminate TLS some time ago because Go's crypto/tls has not been reviewed by 3rdparty security firms. Perhaps that has changed since then. I'd certain like to hear what's @agl's current position.

If Go's TLS implementation still hasn't been critically reviewed from a security standpoint, perhaps the Go team should dedicate some resources to fix the issue as Go is very popular these days and relying on OpenSSL to terminate TLS is silly for a safe programming language with native TLS support.

Merovius commented 8 years ago

@mholt

What do you mean by this? nginx isn't any more "frontend-hardened" than Go is, unless you configure it a bunch. Go is the same way, but it is certainly production-ready.

What exactly is the point in linking to that? If your point is that dl.google.com uses it: Does it, though? @bradfitz knows better, this might well be an exception, but from what I know about the Google Architecture (and that slide I linked to kinda confirms it), almost all services there are sitting behind a reverse proxy, which terminates the TLS connection.

What official communication?

For example this, by one of the central people in implementing it. There have been similar communications before, on go-nuts, in issues, on reddit…

What defines well-vetted?

I'd say that if the people who wrote the tls stack calls it "basically unreviewed", then that's a good indication that, no matter where you draw the line, this doesn't pass it. (or rather: Didn't pass it in the past. All of this might've changed, but that was one of the points of my question: has it changed?)

Go's crypto/tls package has had much fewer vulnerabilities than OpenSSL has

That is a fallacy. Go's crypto/tls package has had much fewer vulnerabilities disclosed than OpenSSL has. It also had much fewer people reading and vetting it (it's used by fewer people, so fewer companies have incentive to vet it and academia has less incentive as the impact is lower). If I were to write a TLS implementation, it would have zero vulnerabilities disclosed. Doesn't make it secure.


All of this, however, really is a tangent. I am not trying to argue that people shouldn't use go's TLS implementation. I'm merely asking whether it is considered secure enough for production use right now, i.e. whether the go team now thinks that it should be used.

adg commented 8 years ago

On 14 September 2016 at 17:07, Axel Wagner notifications@github.com wrote:

whether the go team now thinks that it should be used.

I, for one, think that Go's TLS implementation is suitable for use in production settings.

olliephillips commented 8 years ago

Fabulous proposal. LetsEncrypt has made TLS accessible, but installing and renewing certificates in own TLS server applications isn't as straight forward as for other servers. This is a great idea in my opinion.

gertcuykens commented 8 years ago

I am absolutely in favor of this, most common reason somebody would be against it is because of the fear once it's in, you have to live with it for the rest of your life even if LetsEncrypt get replaced by another company that does it differently. I don't understand why we want to turn golang into cobol, can we not be a little more flexible in this go1.x backward compatible stuff please. Meaning if ACME get deprecated, fine, remove it from standard library in next release, change 3 lines of code recompile, done. To me that 1.x promise is more like a curse than a blessing sometimes.

rsc commented 8 years ago

Given that it's not easy today, why not make it easy with an x/ import first and then reevaluate whether it needs the extra step to go into the standard library? I could see an argument for being in the standard library if it were also going to be on by default, but that doesn't seem right anyway. The x/ repos are supposed to be staging for the standard library (like we did with context) but in this case there has been no staging.

alex commented 8 years ago

Does Go have a policy about the docs for the standard library including examples of things in x/ libraries, or even external libraries? It seems like a great set of first steps is 1) making the API as easy as envisioned here with an acme.Listener, 2) having the net/http docs examples include this; neither of which require the Go team to make a long term commitment.

On Thu, Sep 15, 2016 at 2:07 PM, Russ Cox notifications@github.com wrote:

Given that it's not easy today, why not make it easy with an x/ import first and then reevaluate whether it needs the extra step to go into the standard library? I could see an argument for being in the standard library if it were also going to be on by default, but that doesn't seem right anyway. The x/ repos are supposed to be staging for the standard library (like we did with context) but in this case there has been no staging.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17053#issuecomment-247404878, or mute the thread https://github.com/notifications/unsubscribe-auth/AAADBKmmgsAs9tI_mAzFG-nj_IR3wpMfks5qqYlUgaJpZM4J5z9d .

"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: D1B3 ADC0 E023 8CA6

bradfitz commented 8 years ago

@rsc, @alex, sounds good. I'll start with the autocert.Listener type and some example docs for std.