search

cockroachdb / k8s

Images and utilities to run cockroach on kubernetes
Apache License 2.0
26 stars 25 forks source link

overhauled request-cert. #34

Closed siennathesane closed 1 year ago

siennathesane commented 4 years ago

This brings the request-cert dependencies up to supported versions, and fixes the signing process so the Kubernetes CSR APIs will auto-provision certificates. Currently, the tool does not auto-provision the certificates once they've been approved, and following the installation instructions does not work. I verified this by following the steps in the installation instructions and I can verify the CSRs were provisioned properly and the cluster does start. I also verified the secure client implementation works as well, I've attached a screenshot.

# installed like this.
$ helm install helm install pine-db cockroachdb/cockroachdb --namespace pine --values <<EOF
fullnameOverride: pine-db
statefulset:
  resources:
    limits:
      memory: "8Gi"
    requests:
      memory: "4Gi"
conf:
  cache: "1Gi"
  max-sql-memory: "2Gi"
  locality: "country=us,region=us-east,datacentre=ny7"
storage:
  persistentVolume:
    storageClass: vsan-storage
tls:
  enabled: true
  init:
    image:
      repository: rtseng.azurecr.io/cockroach-k8s-request-cert
      tag: 0.5.0
      pullPolicy: Always
      credentials: {} # omitted for security
EOF

# wait a minute, then approve.
$ kctl get csr | \
        grep "pine" | awk '{print $1}' | \
        xargs kctl certificate approve

# see them issued.
$ kctl get csr
NAME                  AGE   SIGNERNAME                            REQUESTOR                            CONDITION
pine.client.root      28m   kubernetes.io/kube-apiserver-client   system:serviceaccount:pine:pine-db   Approved,Issued
pine.node.pine-db-0   28m   kubernetes.io/legacy-unknown          system:serviceaccount:pine:pine-db   Approved,Issued
pine.node.pine-db-1   27m   kubernetes.io/legacy-unknown          system:serviceaccount:pine:pine-db   Approved,Issued
pine.node.pine-db-2   28m   kubernetes.io/legacy-unknown          system:serviceaccount:pine:pine-db   Approved,Issued

# all the nodes start properly.
$ kctl get pods
NAME                        READY   STATUS      RESTARTS   AGE
cockroachdb-client-secure   1/1     Running     0          19m
pine-db-0                   1/1     Running     0          15m
pine-db-1                   1/1     Running     0          28m
pine-db-2                   1/1     Running     0          28m
pine-db-init-7f8kw          0/1     Completed   0          28m

# the init-certs initContainer runs successfully.
$ kctl describe pods pine-db-0
Name:         pine-db-0
Namespace:    pine
Priority:     0
Node:         eng-beta-worker-3/10.20.70.197
Start Time:   Wed, 05 Aug 2020 11:47:20 -0600
Labels:       app.kubernetes.io/component=cockroachdb
              app.kubernetes.io/instance=pine-db
              app.kubernetes.io/name=cockroachdb
              controller-revision-hash=pine-db-75fd68994b
              statefulset.kubernetes.io/pod-name=pine-db-0
Annotations:  cni.projectcalico.org/podIP: 10.42.5.50/32
              cni.projectcalico.org/podIPs: 10.42.5.50/32
Status:       Running
IP:           10.42.5.50
IPs:
  IP:           10.42.5.50
Controlled By:  StatefulSet/pine-db
Init Containers:
  init-certs:
    Container ID:  docker://827f05da164b7503349d38752f05c561170ced0d3e24492e6090b3b31b54249f
    Image:         rtseng.azurecr.io/cockroach-k8s-request-cert:0.5.0
    Image ID:      docker-pullable://rtseng.azurecr.io/cockroach-k8s-request-cert@sha256:54294e254852fe66939f467aaa2fb4ba1c0d4e3e105e42f1ff5aec1c7d6c1ef6
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/ash
      -ecx
      /request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-certs/ -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt -type=node -addresses=localhost,127.0.0.1,$(hostname -f),$(hostname -f|cut -f 1-2 -d '.'),pine-db-public,pine-db-public.$(hostname -f|cut -f 3- -d '.')
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 05 Aug 2020 11:47:32 -0600
      Finished:     Wed, 05 Aug 2020 11:47:32 -0600
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  pine (v1:metadata.namespace)
    Mounts:
      /cockroach-certs/ from certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from pine-db-token-lqz7b (ro)
Containers:
  db:
    Container ID:  docker://3dfbc97236cac15a9163f0eb7ff4048962399078cc79a0d23d666406a3368eb3
    Image:         cockroachdb/cockroach:v20.1.4
    Image ID:      docker-pullable://cockroachdb/cockroach@sha256:578d1a10d3fb913c2db96c34ac4943a9bcb262babc54d0734b7f9fbfe902618c
    Ports:         26257/TCP, 8080/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      shell
      -ecx
      exec /cockroach/cockroach start --join=${STATEFULSET_NAME}-0.${STATEFULSET_FQDN}:26257,${STATEFULSET_NAME}-1.${STATEFULSET_FQDN}:26257,${STATEFULSET_NAME}-2.${STATEFULSET_FQDN}:26257 --advertise-host=$(hostname).${STATEFULSET_FQDN} --logtostderr=INFO --certs-dir=/cockroach/cockroach-certs/ --http-port=8080 --port=26257 --cache=1Gi --max-disk-temp-storage=0 --max-offset=500ms --max-sql-memory=2Gi --locality=country=us,region=us-east,datacentre=ny7
    State:          Running
      Started:      Wed, 05 Aug 2020 11:47:32 -0600
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  8Gi
    Requests:
      memory:   4Gi
    Liveness:   http-get https://:http/health delay=30s timeout=1s period=5s #success=1 #failure=3
    Readiness:  http-get https://:http/health%3Fready=1 delay=10s timeout=1s period=5s #success=1 #failure=2
    Environment:
      STATEFULSET_NAME:   pine-db
      STATEFULSET_FQDN:   pine-db.pine.svc.cluster.local
      COCKROACH_CHANNEL:  kubernetes-helm
    Mounts:
      /cockroach/cockroach-certs/ from certs (rw)
      /cockroach/cockroach-data/ from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from pine-db-token-lqz7b (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-pine-db-0
    ReadOnly:   false
  certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  pine-db-token-lqz7b:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pine-db-token-lqz7b
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From                        Message
  ----    ------     ----       ----                        -------
  Normal  Scheduled  <unknown>  default-scheduler           Successfully assigned pine/pine-db-0 to eng-beta-worker-3
  Normal  Pulling    18m        kubelet, eng-beta-worker-3  Pulling image "rtseng.azurecr.io/cockroach-k8s-request-cert:0.5.0"
  Normal  Pulled     18m        kubelet, eng-beta-worker-3  Successfully pulled image "rtseng.azurecr.io/cockroach-k8s-request-cert:0.5.0"
  Normal  Created    18m        kubelet, eng-beta-worker-3  Created container init-certs
  Normal  Started    18m        kubelet, eng-beta-worker-3  Started container init-certs
  Normal  Pulled     18m        kubelet, eng-beta-worker-3  Container image "cockroachdb/cockroach:v20.1.4" already present on machine
  Normal  Created    18m        kubelet, eng-beta-worker-3  Created container db
  Normal  Started    18m        kubelet, eng-beta-worker-3  Started container db

# for reference.
$ kctl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-07-06T19:16:02Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

image

There are still some concerns I've raised in cockroachdb/cockroach#52418, so if that gets implemented, then applying the fix here is now trivial.

Signed-off-by: Mike Lloyd mlloyd@rts.com

jrote1 commented 4 years ago

@mxplusb I just tried building this PR and got the following error, are one of the dependencies not constrained?

#12 6.902 go: finding module for package github.com/googleapis/gnostic/OpenAPIv2
#12 7.176 /go/pkg/mod/k8s.io/client-go@v0.18.6/discovery/discovery_client.go:30:2: module github.com/googleapis/gnostic@latest found (v0.5.1), but does not contain package github.com/googleapis/gnostic/OpenAPIv2

Looks like it could be related to https://github.com/googleapis/gnostic/issues/156

Update: I have confirmed that running go get github.com/googleapis/gnostic@v0.4.0 fixes the build issue

siennathesane commented 4 years ago

@jrote1 it looks like it was a transitive dependency, I locked the dependency and pushed, should be good.