Secure etcd by default

[gyuho]

• Update all start-up scripts or start guide docs with TLS or auth.
• Never allow unauthenticated client access by default (either by auth or TLS)?
• Enable TLS by default for peer-to-peer connection https://github.com/coreos/etcd/issues/4022?

/cc @philips @ericchiang

[hexfusion]

@gyuho is this a 4.0 proposal or 3.4+ Although I think it is a great step forward for security it will create a storm of breakage for insecure tests, deployments etc.

[gyuho]

@hexfusion Probably, we need research more!

[ericchiang]

Update all start-up scripts or start guide docs with TLS or auth.

+1

Never allow unauthenticated client access by default (either by auth or TLS)?

The hard part about this is that users are still required to supply client-auth flags. So they're still going to either generate TLS certs and add new flags or just supply --insecure-listen-http.

If we're punting this to 4.0, I think it'd be better to make it impossible to start an etcd server that allowed unauthenticated requests

[rugk]

You got a good timing: https://arstechnica.com/information-technology/2018/03/thousands-of-servers-found-leaking-750-mb-worth-of-passwords-and-keys/

I hope TLS here refers to some TLS-auth with client certs, as otherwise it is of course no authentication.

[philips]

Can someone research what other databases do? I am skeptical we want to turn authentication mechanisms on by default as it increases friction to the dev/test use case and then the natural question is what is the "default" authentication. For example do you generate a random password, generate a TLS cert, etc and force that to happen on first run? And then where do you put that? In a log file, put it on disk, etc?

@ericchiang This isn't about insecure HTTP but instead about mTLS or some other authentication method.

[hexfusion]

Can someone research what other databases do

This will get the list started.

MySQL 5.7.x:

• Default is a root user with a random password which is written to the error log.

  2016-05-16T07:47:58.199154Z 1 [Note] A temporary password is generated for root@localhost: wzgds/:Kf2,g

ref: Percona

---

MongoDB 3.6.x

• Default is no auth listening to localhost.

ref: MongoDB

---

CockroachDB 1.1.x

• Default is to require a TLS cert dir containing cert, key and a separate ca dir. But you can pass --insecure flag to bypass.

ref: CockroachDB

[philips]

@hexfusion

• MySQL: seems bad since it it reduces automation for simple dev/test and puts credentials into logs
• MongoDB: etcd does exactly this today: listen on localhost no auth
• Cockroachdb: seems reasonable but I don't understand how much value --insecure adds over the default to only listen on localhost. If users are that silly I don't know how to help them. Is there evidence that a --insecure flag actually is enough of a speedbump?

[hexfusion]

I guess it is a fair point that there is not much technically different between

--listen-client-urls http://10.0.1.10:2379,http://127.0.0.1:2379

--insecure

as they are both actions taken by the admin to change defaults in a possibly insecure manner.

But IMO there is a difference between requiring full chain TLS by default and listening to localhost. The message and intention to promote secure usage is much more obvious and clear. We can not control the users but we can define our vision of what should be the best practice as a default. Setting ETCD_INSECURE=1 would not be very obtrusive for testing, and can not get any more obvious to the user.

[c-schmitz]

This issue made it to the news: https://developers.slashdot.org/story/18/03/24/1943239/shodan-search-exposes-thousands-of-servers-hosting-passwords-and-keys

[mitake]

@yudai shared a problem related to this issue with me. He's caring about the case of failed initialization scripts. Having such a script (e.g. configuring users, roles and permission then enable auth) for initializing an etcd cluster is usual. If the script fails before enabling auth because of some reasons e.g. bugs or resource shortages, the cluster will be running without auth. In such a case, malicious users can create keys.

In his case managing TLS certs is too costly. So adding an option which enables auth at startup time to etcd would be valuable (the option won't change the default behaviour).

[peebs]

Maybe allowing a symmetric secret for authenticated encryption could simplify secret mgmt for the base case of simple deployments and testing? Two strings of secret state, one secret for peer-to-peer, one secret for client-auth, both can be passed via flags or env.

Simply using aes-gcm seems OK for this mode, many other options possible.

[ericchiang]

@pbx0 if you can place a symmetric key you can place TLS assets.

I think the overhead of supporting a form of transport security other than TLS is going to be way more than just providing scripts to generate TLS assets.

[peebs]

I think having two strings to deal with just makes the base case so much more simple to deal with compared to : https://coreos.com/etcd/docs/latest/op-guide/security.html

The technical overhead may not worth supporting and thats fair, but I think it may be worth evaluating. Entering two easily copy-pasted flags vs a collection of files, secrets and flags just seems radically more understandable and easier for simple first-time setups.

Edit: also worth making it clear that one could no longer simply curl keys which could be a big deal if someone didn't realize that this mode limited your client options (maybe there is a way to do symmetric tls that curl understands?)

[raoofm]

@pbx0 the problem with symmetric keys is they can't be rotated. This option has been discussed and rejected a lot of times and initially during the design phase of etcd.

I also agree with @ericchiang

[philips]

Aside: I wish TLS PSK was merged into Go

[peebs]

@raoofm I'm not sure all use cases require rotation which can also be complicated. I'm only suggesting that maybe there is room in the design space for a more limited, less flexible mode of security that is far easier to use and get up and running, the standard TLS setup would remain available.

@philips Agreed, it'd certainly make this crazy idea a lot more tolerable from a implementation and maintenance point of view.

[heyitsanthony]

Maybe this is obvious / won't work, but a unique initial symmetric key / root password could be derived from the initial cluster configuration (cluster id, hash of members, etc). An offline utility would get the key by processing data from the member dir / boltdb.

[rugk]

So if you know the configuration, you know the key? That does not sound secure to me at all. Especially as it could actually be counterproductive, because users could think that the key is secure and they don't need to change it, etc. Then rather just generate a random key and leave it as that.