search

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.83k stars 9.77k forks source link

Use of client's crt/key pair to launch a cluster memeber #8607

Closed ayashjorden closed 7 years ago

ayashjorden commented 7 years ago

Hello ETCd team, I've got a security question for you.

Assuming that we have a TLS enabled cluster with N members. Both --peer-client-cert-auth and --client-cert-auth are enabled as well.

Given that one gains access to a client's TLS crt/key pair, will he/she be able to launch a cluster member ETCd instance using the client's TLS crt/key pair and it will successfully join the cluster?

I've read through #8262, which is great, but currently I have to run two CAs, one will generate ETCd cluster peers' crt/key-s and the second will generate clients' crt/key-pairs (k8s, applications and such.)

The issue is not clear on the TLS page.

I will be able to test it in a couple of days, but until then, if anyone can comment or share experience, that would be great 👍

Best, Yarden

jpbetz commented 7 years ago

@ayashjorden Are you asking if the cert/key pairs used for "client-to-server" communication can also be used for "server-to-server / cluster" communication? My reading of the TLS page is that they are separate. What did you find unclear about the page?

ayashjorden commented 7 years ago

Hi @jpbetz,

My reading of the TLS page is that I can use two crt/key pairs in the cluster-member configuration. One for server-to-server and one for client-to-server. but it does not explain how cluster members are controlled.

Given that I have one CA that issues certificates for cluster-members and clients(with CN or SAN), What prevents me from using a client-to-server crt/key pair for server-to-server / cluster communication?

Is there some RBAC for cluster members?

gyuho commented 7 years ago

will he/she be able to launch a cluster member ETCd instance using the client's TLS crt/key pair and it will successfully join the cluster?

Only if your server certs have matching hosts (either exact or wildcard matching). DNS SAN wildcard reverse lookup is only available from etcd v3.2.5.

ayashjorden commented 7 years ago

That would be a nice experiment

As @Ulexus mentioned in this thread, the best practice for preventing interloop cluster members is multiple CAs. one for server-to-server and another for client-to-server.

IMO, this information/scenario should be mentioned-explained in ETCd security page.

Any comments/ideas?

Ulexus commented 7 years ago

Also note that the PKI for the members is only used between members, so it's much easier (better?) to use a private, unaffiliated, unregistered, undistributed CA for the peering system.

You can (should) create your client certificates such that they do not have any authorized IPs and thus cannot match any host, causing them to fail their CN and SAN checks, which will cause them to be unable to peer. However, you're really at the mercy of any number of problems anyway. A client could change any data in the database. A client could change the membership such that quorum is lost. A client could add members which bypass non-source-discriminatory firwalls.

Basically, then, if you can't trust your clients, you really need to use role-based access control on the client side. No amount of PKI wrangling alone will protect your cluster if your clients are compromised.

ayashjorden commented 7 years ago

@Ulexus, to clarify, by

you really ned to use RBAC on the client side

you mean the ETCd RBAC system on the KeySpace?

I'm curious how a client can issue membership/quorum related commands?

And even more, is there a page/article about the KeySpace structure? (not that its a B-tree, but the functional structure... how metadata is stored, cluster and clients' data?

Ulexus commented 7 years ago

The root roles provide access to the cluster membership controls, from what I understand, yes. In etcdv2, the membership data was stored inside the key space like everything else. I presume (but have not verified) that this is still true for etcdv3.

If the vulnerability you are trying to address is a compromised client keyset (and, one presumes, any authentication thereof), then keeping your operational clients constrained within restricted roles seems to be the surest answer.

ayashjorden commented 7 years ago

Thank you all for the valuable info and insights.

Is there any way I can help to get this information to the community in a more visible way than a Github issue?

gyuho commented 7 years ago

We have auth doc https://github.com/coreos/etcd/blob/master/Documentation/op-guide/authentication.md. And https://github.com/coreos/etcd/blob/master/Documentation/op-guide/security.md. If you find anything missing, please let us know. Thanks!

ayashjorden commented 7 years ago

@gyuho, The two links provided does not describe what what discussed in this ticket about best-practices for authorization of cluster-members vs. cluster-clients.