request-cert type client

[gertcuykens]

request-cert type node works but type client give me the following error

   initContainers:
      - name: init-certs
        image: cockroachdb/cockroach-k8s-request-cert:0.4
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: certs
          mountPath: /cockroach-certs
        - name: client
          mountPath: /cockroach-client
        command:
        - "/bin/ash"
        - "-xec"
        # - "/request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-certs -type=node -addresses=localhost,127.0.0.1,$(hostname -f),$(hostname -f | cut -f 1-2 -d '.'),cockroach-lb,cockroachdb-lb.$(hostname -f | cut -f 3- -d '.') -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt;"
        - "/request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-client -type=client -user=root -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"

% kubectl get csr
NAME                       AGE       REQUESTOR                                 CONDITION
default.client.root        18m       system:serviceaccount:default:cockroach   Approved,Issued
default.node.cockroach-0   51m       system:serviceaccount:default:cockroach   Approved,Issued
default.node.cockroach-1   48m       system:serviceaccount:default:cockroach   Approved,Issued
default.node.cockroach-2   48m       system:serviceaccount:default:cockroach   Approved,Issued

+ /request-cert '-namespace=default' '-certs-dir=/cockroach-client' '-type=client' '-user=root' '-symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2018/10/29 04:06:51 Looking up cert and key under secret default.client.root
W1029 04:06:51.502966       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018/10/29 04:06:51 Secret default.client.root not found, sending CSR
Sending create request: default.client.root for 
2018/10/29 04:06:51 failed to get certificate: CertificateSigningRequest.Create(default.client.root) failed: certificatesigningrequests.certificates.k8s.io "default.client.root" already exists

PS Not related to the problem but what's the best way to combine node and client cmd into one init cmd? Tried ; but that doesn't work for me.

[a-robinson]

Hey @gertcuykens, sorry for the delay! I've been out for a couple weeks.

Is this readily reproducible? I could see things getting into this scenario if some init container created a CSR, then got killed before it was approved, then the admin approved the certificate. It's certainly a failure of the code here if it doesn't handle that scenario (it should be happy to just grab the signed cert), but I want to know if that's all the happened or if there's something more fundamental that went wrong here.

I can't see why it would, but I also wonder if changing the directory to -certs-dir=/cockroach-client is causing problems.

[gertcuykens]

Yes I think I didn't do anything special, unfortunalty I dont have a kubenernets available to dig into. Pretty sure it will come up again, let just wait until somebody else encounters it too

[mstg]

Just to note: deleting the CSR with kubectl delete csr <name> and retrying then approving fixes the issue. I've kinda wasted embarrassingly long time on how to overcome this problem even though it was very obvious so I hope anyone else seeing this saves some time :smile: