search

kz8s / tack

Terraform module for creating Kubernetes cluster running on Container Linux by CoreOS in an AWS VPC
MIT License
719 stars 145 forks source link

Certificate rejected with Etcd 3.2.9 and 3.3.0-rc.2 #213

Open novinfor opened 6 years ago

novinfor commented 6 years ago

Hi,

I'm having the following issues with Etcd certificate generated by Tack (cfssl).

make all fails on ❤ Curling apiserver external elb.

Cluster was started with Etcd version 3.2.9 which reports the following errors:

Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378578 I | etcdmain: etcd Version: 3.2.9
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378592 I | etcdmain: Git SHA: f1d7dd8
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378602 I | etcdmain: Go Version: go1.8.4
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378611 I | etcdmain: Go OS/Arch: linux/amd64
...
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378724 I | embed: peerTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = false
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.471926 I | embed: ClientTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = true
...
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.618877 I | etcdmain: rejected connection from "10.0.10.11:54576" (remote error: tls: bad certificate)
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.628363 I | etcdmain: rejected connection from "10.0.10.12:50336" (remote error: tls: bad certificate)

After enforcing the Etcd latest version by adding the ETCD_IMAGE_TAG environment variable to modules/etcd/cloud-config.yml and modules/worker/cloud-config.yml:

Environment="ETCD_IMAGE_TAG=v3.3"

The error message changed slightly:

Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256196 I | etcdmain: etcd Version: 3.3.0-rc.2
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256208 I | etcdmain: Git SHA: 9e079d8f0
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256217 I | etcdmain: Go Version: go1.9.2
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256225 I | etcdmain: Go OS/Arch: linux/amd64
...
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256335 I | embed: peerTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = false, crl-file =
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.340613 I | embed: ClientTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = true, crl-file =
...
Jan 15 12:05:12 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:12.656516 I | embed: rejected connection from "10.0.10.11:42408" (error "remote error: tls: bad certificate", ServerName "test.kz8s")
Jan 15 12:05:12 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:12.664169 I | embed: rejected connection from "10.0.10.12:51198" (error "remote error: tls: bad certificate", ServerName "test.kz8s")

but the problem persists :(

Etcd version 3.1.11 (ETCD_IMAGE_TAG=v3.1.11) is working fine and cluster is started successfully.

Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529288 I | etcdmain: etcd Version: 3.1.11
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529299 I | etcdmain: Git SHA: 960f460
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529307 I | etcdmain: Go Version: go1.8.5
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529315 I | etcdmain: Go OS/Arch: linux/amd64
...
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529389 I | embed: peerTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = false
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.617281 I | embed: ClientTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = true
...
Jan 15 11:40:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:10.339317 I | embed: ready to serve client requests

Versions used:

❤ checking prerequisities

# aws --version
aws-cli/1.11.13 Python/3.5.2 Linux/4.9.49-moby botocore/1.4.70

# cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6.2

# jq --version
jq-1.5-1-a5b5cbe

# kubectl version --client
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

# terraform --version
Terraform v0.11.2

and Tack 10bc698.

Thanks for checking this issue.

jacohend commented 6 years ago

@novinfor I'm getting this with kube-aws too...I wonder if this is an etcd issue, or at least somewhere upstream of this?