search

coreos / tectonic-docs

Tectonic documentation - https://coreos.com/tectonic/docs/latest/
Apache License 2.0
42 stars 49 forks source link

tls: add scripts and instructions for rotating certificates #155

Closed ericchiang closed 6 years ago

ericchiang commented 6 years ago

This is a revamp of our TLS rotation docs. I've been testing them on more recent clusters (1.8.x) on AWS.

Etcd rotation instructions will be added in a bit, but I'd like early feedback.

@kbrwn for testing @robszumski for general review @zbwright for docs

ericchiang commented 6 years ago

Some open questions:

ericchiang commented 6 years ago

It was pointed out that kube-proxy reuses the kubelet's certs https://github.com/coreos/tectonic-installer/blob/1.8.9-tectonic.1/modules/bootkube/resources/manifests/kube-proxy.yaml#L31

Need to roll that daemonset as well.

ericchiang commented 6 years ago

This document is almost ready to merge, but has a bug in it that keeps bricking my clusters...

I cannot stress this warning at the beginning more

__WARNING:__ Rotating certificates by hand can break component connectivity and leave the cluster in an unrecoverable state. Before performing any of these instructions on a live cluster backup your cluster state and migrate critical workloads to another cluster.
ericchiang commented 6 years ago

This is done. @zbwright would you take a look one last time?

justaugustus commented 6 years ago

@ericchiang Do you have any context on why the clusters are getting bricked? A warning like that seems super troubling.

ericchiang commented 6 years ago

If you mess up the etcd CA rotation, then it's really hard to change anything on the self hosted control plane. I can expand on that in the doc.

On Fri, Apr 6, 2018, 3:21 PM Stephen Augustus notifications@github.com wrote:

@ericchiang https://github.com/ericchiang Do you have any context on why the clusters are getting bricked? A warning like that seems super troubling.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/coreos/tectonic-docs/pull/155#issuecomment-379400117, or mute the thread https://github.com/notifications/unsubscribe-auth/ACO_XWQxDZ4jWE5cRs58CXBnFf0g3m_sks5tl-pJgaJpZM4TAyzg .

ericchiang commented 6 years ago

To be clear that bug I mentioned earlier was resolved, but still I'd tread very carefully here.

justaugustus commented 6 years ago

@ericchiang cool, cool. Thanks for the clarification!

ericchiang commented 6 years ago

bumping this thread. there was some interest in more testing beside's me just doing it. did that ever get planned/done?

justaugustus commented 6 years ago

@ericchiang I was hope to get some field validation on this, but as it's only Dan (and me, in a diminished capacity), I don't think we can commit to any testing in the near-term, so don't block this on my account.

@kbrwn mentioned he was working with someone, but would need to check-in again to try out the etcd rotation.

ericchiang commented 6 years ago

Docs updated.

ericchiang commented 6 years ago

Okay it's been a bit. I'm merging this tomorrow afternoon unless someone says otherwise.