kubectl exec throws tls error

[sanjana-bhat]

Hi,

I have been getting the following error when I use kubectl exec command. kubectl exec -p nimbus-controller-hqziy date error: Unable to upgrade connection: { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "tls: oversized record received with length 20527", "code": 500 }

I'm not using any CertFile or CAFile. Does exec always perform authentication? If so, does k8sm support any flags to skip this? I'm using the upstream_k8sm branch to build k8sm.

Thanks, Sanjana

[jdef]

It looks like they're attempting to upgrade from HTTP to SPDY to support bi-directional transfer: https://github.com/GoogleCloudPlatform/kubernetes/blob/release-0.19/pkg/client/remotecommand/remotecommand.go#L98

The upgrade proc is here (looks like it requires TLS, which k8s-mesos doesn't currently support): https://github.com/GoogleCloudPlatform/kubernetes/blob/release-0.19/pkg/client/request.go#L626

[jdef]

Actually, I shouldn't have said "... which k8s-mesos doesn't currently support". It looks to me like the connection is being made to the apiserver, which is a stock k8s component. So it may work if you run the apiserver and kubectl with TLS configured appropriately.

[jdef]

@sanjana-bhat Out of curiosity, are you doing this on DCOS or on some other mesos cluster?

[sanjana-bhat]

@jdef, we don't use DCOS cluster. I will take a look at configuring TLS. Do you have any pointers for this?

[jdef]

Nevermind, I see that you indicated upstream_k8sm. BTW that branch is pretty unstable. I actually just removed all of the Go code and build files. If you want the latest k8s-mesos stuff it now lives upstream in the k8s repo. There are some pointers in the k8s-mesos docs about how to build (e.g. DEVEL.md). We'll eventually move the docs upstream as well.

Regarding TLS: I think the apiserver lets you specify a cert and key file at startup (--tls* params). If you don't, it generates what it needs (you should be able to see this in the apiserver logs). kubectl also has --certificate* and --client-* options that you'll probably need to set. I haven't played with it too much so I can't provide more advice here.

[sanjana-bhat]

@jdef, the reason we were using upstream_k8sm is because of https://github.com/mesosphere/kubernetes-mesos/issues/301

Do I have to refer to this for setup steps? https://github.com/mesosphere/kubernetes-mesos/blob/master/DEVELOP.md

[jdef]

Yeah, but the upstream_k8sm version of that, which basically says to:

export KUBERNETES_CONTRIB=mesos

... and then follow the k8s guide for building k8s binaries by hand. In practice, this amounts to cloning the upstream repo and running make. The above env var is needed to tell k8s to also build the k8s-mesos binaries.

[sanjana-bhat]

@jdef, thank you for clarifying the setup steps. Does this mean you will be deprecating kubernetes-mesosphere git repo and all your changes will go under the upstream repo from now on? Should we continue using the upstream k8s repo and not depend on kubernetes-mesosphere repo?

[jdef]

Yes, that's correct: all future code changes will happen upstream.

[sanjana-bhat]

Great! Thanks for your help.

[jdef]

re-opening this so that we don't lose track of it. we'll want to make sure that this works with DCOS

[jdef]

TODO: re-test this now that initial TLS support has landed

[huang195]

This seems to me to be not TLS related. I tested this out using the HEAD of upstream/kubernetes in a K8sm environment (using mesos 0.27)

root@rsch-dal09-haih-k8sm-1:~/k8sm-scripts# kubectl --server=http://10.143.100.209:8080 exec -c redis guestbookcontroller-19l1w date
Thu Feb 25 16:07:25 UTC 2016
root@rsch-dal09-haih-k8sm-1:~/k8sm-scripts# kubectl --server=http://10.143.100.209:8080 exec -c guestbook guestbookcontroller-19l1w date
Thu Feb 25 16:08:12 UTC 2016

I did not configure TLS at all, and all the communications are through insecure ports.

[sttts]

kubectl exec through insecure ports does not work. It needs SPDY on top of https.

[huang195]

It seems to have worked for me. Had to configure nginx (in front of multiple apiservers) to handle spdy upgrade, but once that's done, kubectl exec through nginx worked without tls. Just to verify, kubectl exec bypassing nginx and going directly to apiserver worked as well.