"No client certificate CA names sent"

[jimmidyson]

Loving this project btw!

One problem I'm having is reusing the client certificate that minikube uses ootb. I don't think the API server is properly requesting client CA names for optional client certificates, even if client-ca-file flag is used.

Here's the command I'm using to start the API server (which works when using the bearer token provided btw):

$ ./apiserver --authentication-kubeconfig ~/.kube/auth_config \
  --authorization-kubeconfig ~/.kube/auth_config \
  --client-ca-file /home/jdyson/.minikube/ca.crt \
  --requestheader-client-ca-file /home/jdyson/.minikube/ca.crt \
  --requestheader-username-headers=X-Remote-User \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --etcd-servers=http://localhost:2379 \
  --secure-port=9443 \
  --tls-ca-file ./apiserver.local.config/certificates/apiserver.crt

and then running:

$ kubectl get integrations

Error from server (Forbidden): User "system:anonymous" cannot list <kind>.<apigroup> in the namespace "default". (get <kind>.<apigroup>)

Using openssl to see if the client cert CA name was sent to the client shows it wasn't:

$ openssl s_client -state -connect localhost:9443 -CAfile -CAfile ./apiserver.local.config/certificates/apiserver.crt

CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=0 CN = localhost@1495047152
verify return:1
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read server session ticket A
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/CN=localhost@1495047152
   i:/CN=localhost@1495047152
---
Server certificate
<SNIP>
subject=/CN=localhost@1495047152
issuer=/CN=localhost@1495047152
---
No client certificate CA names sent
Peer signing digest: SHA384
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1396 bytes and written 434 bytes
---
<SNIP>

Notce the No client certificate CA names sent line - assuming that is the problem?

[pwittrock]

Hm, I don't know a whole lot about this. My plan has been to wait until apiserver aggregation works ootb in 1.7 and minikube is using 1.7 before digging in too deeply to the integration. However if you find a fix in the mean time would be happy to take it.

[jimmidyson]

Don't suppose you've any idea where to start looking? in kubernetes/apiserver I guess?

[pwittrock]

Hm. Maybe start by running kubectl -v and seeing if it is requesting the proper headers or what it is doing. Then I would try a different canonical version of Kubernetes (GCE using the up scripts?), and see if the apiserver works properly with that cluster. I think I have gotten this working with GCE before. Minikube runs in a vm and is setup a little differently than a standard cluster IIRC, so I would eliminate that as an issue. Let me know what information you get out of that, and I will see if I know enough to help further.

[DirectXMan12]

agreed -- I have a bit of experience with this stuff setting it up for service catalog, Heapster, custom metrics, etc.

Seeing exactly what kubectl is spitting out, as well as what the logs from the API server are, would be quite useful. If that's not helpful, I can try and make sure everything's set up correctly in the generated code.

[jimmidyson]

So here's the kube config being used by kubectl:

$ kubectl config view --minify

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/jdyson/go/src/github.com/jimmidyson/test-apiserver/apiserver.local.config/certificates/apiserver.crt
    server: https://localhost:9443
  name: test
contexts:
- context:
    cluster: test
    user: minikube
  name: test
current-context: test
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /home/jdyson/.minikube/apiserver.crt
    client-key: /home/jdyson/.minikube/apiserver.key

And the output from kubectl:

$ kubectl get myresources -v=10

I0523 10:08:20.972649   22231 loader.go:354] Config loaded from file /home/jdyson/.kube/config
I0523 10:08:20.973817   22231 cached_discovery.go:118] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/servergroups.json
I0523 10:08:20.973895   22231 cached_discovery.go:71] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/myresource.test.io/v1alpha1/serverresources.json
I0523 10:08:20.973981   22231 cached_discovery.go:118] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/servergroups.json
I0523 10:08:20.974032   22231 cached_discovery.go:71] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/myresource.test.io/v1alpha1/serverresources.json
I0523 10:08:20.974066   22231 cached_discovery.go:118] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/servergroups.json
I0523 10:08:20.974096   22231 cached_discovery.go:71] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/myresource.test.io/v1alpha1/serverresources.json
I0523 10:08:20.974155   22231 cached_discovery.go:118] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/servergroups.json
I0523 10:08:20.974188   22231 cached_discovery.go:71] returning cached discovery info from /home/jdyson/.kube/cache/discovery/localhost_9443/myresource.test.io/v1alpha1/serverresources.json
I0523 10:08:20.974582   22231 round_trippers.go:398] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/v1.6.2 (linux/amd64) kubernetes/477efc3" https://localhost:9443/apis/myresource.test.io/v1alpha1/namespaces/default/myresources
I0523 10:08:20.979884   22231 round_trippers.go:417] GET https://localhost:9443/apis/myresource.test.io/v1alpha1/namespaces/default/myresources 403 Forbidden in 5 milliseconds
I0523 10:08:20.979897   22231 round_trippers.go:423] Response Headers:
I0523 10:08:20.979901   22231 round_trippers.go:426]     Content-Type: text/plain
I0523 10:08:20.979904   22231 round_trippers.go:426]     X-Content-Type-Options: nosniff
I0523 10:08:20.979907   22231 round_trippers.go:426]     Content-Length: 100
I0523 10:08:20.979910   22231 round_trippers.go:426]     Date: Tue, 23 May 2017 09:08:20 GMT
I0523 10:08:20.979927   22231 request.go:991] Response Body: User "system:anonymous" cannot list myresources.myresource.test.io in the namespace "default".
I0523 10:08:20.980027   22231 helpers.go:206] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "User \"system:anonymous\" cannot list myresources.myresource.test.io in the namespace \"default\". (get myresources.myresource.test.io)",
  "reason": "Forbidden",
  "details": {
    "group": "myresource.test.io",
    "kind": "myresources",
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "User \"system:anonymous\" cannot list myresources.myresource.test.io in the namespace \"default\"."
      }
    ]
  },
  "code": 403
}]
F0523 10:08:20.980044   22231 helpers.go:119] Error from server (Forbidden): User "system:anonymous" cannot list myresources.myresource.test.io in the namespace "default". (get myresources.myresource.test.io)

[jimmidyson]

It's the User \"system:anonymous\" bit which indicates to me that the CN from the client cert isn't being used as the username. I'm assuming that's because the client cert isn't being sent at all, as the preview openssl s_client snippet indicated.

[jimmidyson]

Found it... PR coming... The problem is here which was overwriting the client CA stuff previously set up. Just going to change the order we apply the config in https://github.com/kubernetes-incubator/apiserver-builder/blob/master/pkg/cmd/server/start.go#L151-L170

[pwittrock]

Woot. Thanks.

On May 23, 2017 6:55 AM, "Jimmi Dyson" notifications@github.com wrote:

Found it... PR coming... The problem is here https://github.com/kubernetes-incubator/apiserver-builder/blob/master/vendor/k8s.io/apiserver/pkg/server/options/serving.go#L229 which was overwriting the client CA stuff previously set up. Just going to change the order we apply the config in https://github.com/kubernetes- incubator/apiserver-builder/blob/master/pkg/cmd/server/start.go#L151-L170

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes-incubator/apiserver-builder/issues/37#issuecomment-303405529, or mute the thread https://github.com/notifications/unsubscribe-auth/AHhIAm16PB2e2VJJ67kIZKcU1hi4fFozks5r8uU7gaJpZM4NeZZD .