Security: don't bind to insecure address by default

[tuxcanfly]

By default, microk8s binds to 0.0.0.0, which can lead to security issues. suppoie is a malware which seems to exploit this. Please consider removing the offending arg --insecure-bind-address=0.0.0.0.

[ktsakalozos]

Thank you for reporting this @tuxcanfly. We already have a PR that works towards securing microk8s https://github.com/ubuntu/microk8s/pull/88. It should hit the edge channel soon.

[lmakarov]

I learned about this project at KuberCon last week and started playing with it on a cloud instance.

The instance got exploited within hours...

Since this project is focused on helping developers and novice users getting started with K8s, it would be good to have some bare minimum security to protect users from the obvious attack vectors with K8s.

[StephanX]

Just my two bits, but I'd say the opposite: if users need to secure access to this local development tool, they should learn how to do so. Anyone who's ever tried to kick the tires on redis knows the pain involved when you use a bind address of (say) 127.0.0.1 instead of 0.0.0.0.

[Cryptophobia]

Thank you for the article @lmakarov . I've never tried microk8s but your article is making a great criticism about using --insecure-bind-address=0.0.0.0 by default.

@StephanX that's not really a good excuse. There are lots of tools out there that don't expose and bind directly to 0.0.0.0 by default. Redis always binds automatically to the local interface by default as defined in redis.conf. With Redis you are dealing with one component which can have a sufficiently large enough password to prevent exploitation from remote attackers given you secure the port. With K8s you are dealing with a whole API layer of multiple components usually connected to dns and networking components, which can have even more attack vectors.

An experimental tool that is supposed to be used for development and learning k8s should not serve as an attack vector. Especially not to the new users of k8s. Canonical is going around conferences spreading unsecured software and making it easier for new users to get exploited.

There is probably a better way. Probably --insecure-bind-address=127.0.0.1 for a start.

[StephanX]

@cryptophobia - well, username checks out.

The decision ultimately trades usability for security, as is often the case. The question would be how many issue posts complaining about lack of security become issue posts complaining about usability. It's infinitely easier to point fingers and chide others, than to actually do the work and carry the water.

Instead of wagging fingers, why haven't either of you filed pull requests?

[Cryptophobia]

@Cryptophobia - well, username checks out.

Don't ridicule my name. People can't pick their names or their parents. My parents named me Cryptophobia for a reason and I can't change that.

The decision ultimately trades usability for security, as is often the case. The question would be how many issue posts complaining about lack of security become issue posts complaining about usability.

This trade-off is clearly never outlined in the design documents or documentation. If Canonical is so interested in k8s and making developers lives better with this open source project, then they should at least outline the security risks of running microk8s in any environment or write documentation to secure that environment from the start.

It's infinitely easier to point fingers and chide others, than to actually do the work and carry the water. Instead of wagging fingers, why haven't either of you filed pull requests?

All I'm saying is before marketing people are sent around public conferences hustling products exploited by common malware and asking users to try them, maybe there should be at least better documentation or clearly outlined limitations/security risks.

Basic security from the start is a basic right for users. That's what I believe. Anything less is arrogant and disrespectful to the users.

All in all, this leads me to the conclusion that this repo is just an unfinished product to show off Canonical's snapcraft.io package manager by some Architecture Astronauts. No focus on basic security. Ton of marketing. And adds no value to the open source k8s community.

Honestly, I was going to try microk8s and this snapcraft.io package manager but after reading @lmakarov's article, it is probably not worth my time.

[gtrak]

There's not much reason or benefit of a default 0.0.0.0 bind for software that is intended to be used only for local development. It's not security vs usability, since the intended use-case is clear.

[gbevan]

from the website

... Use it for offline development, prototyping, testing, or use it on a VM as a small, cheap, reliable k8s for CI/CD. Makes a great k8s for appliances - develop your IoT apps for k8s and deploy them to MicroK8s on your boxes.

(i added bold) I'd like to use microk8s as an "appliance" (e.g. for CI/CD pipelines for small teams), but I would like it to be secure enough for (internal) production. That probably means the api binding to 0.0.0.0 but https with certificates for client api connections. I'd welcome your thoughts?

[Cryptophobia]

The default should be secure but also configurable - localhost. Then it should provide the option to bind to 0.0.0.0 or port forward but only at the user's discretion.

@gbevan You will need something like Kong or Keycloak in front of microk8s so that your developers can auth and https for the client api connections to the backend microk8s.

https://ncarlier.gitbooks.io/oss-api-management/content/howto-kong_with_keycloak.html

Or if you do not need auth but only https then you can deploy it behind a proxy like nginx or HAproxy: https://github.com/ubuntu/microk8s#deploy-behind-a-proxy

[ktsakalozos]

Hi,

Thank you all for your input. Those of you who had a bad user experience please accept my appologies.

We all want to have this issue resolved; to do so we need to move forward with PR https://github.com/ubuntu/microk8s/pull/88

Here is what you can do:

• Review the document https://github.com/ubuntu/microk8s/blob/feature/access/docs/ports.md describing the setup porposed. This doc is part of the PR so you can leave your comments there.
• A build with the proposed security fixes is available for you to deploy and test. You need to remove your current MicroK8s and then sudo snap install microk8s --classic --channel=1.13/edge/security-testing
• You can provide a code review on the above PR. Your feedback is much appreciated.

@gbevan I think the proposed patch does exactly what you describe. The API service is available on port 16443 over SSL and users have to present proper credentials.

@gtrak, @Cryptophobia thank you for your input, I would appreciate any feedback you may give on the proposed patch.

@StephanX, thank you for presenting the low-security approach. I totally feel your frustration everytime I fight with the dev tools/frameworks instead of focusing on the problem I have to solve. I would really appreciate if you could deploy MicroK8s from channel 1.13/edge/security-testing and let me know if the user experience is much worse.

@lmakarov, thank you for the blog, it was a nice read. The security issues are always a concern. I have to thank you, because by speaking out loud you give us the oportunity to improve MicroK8s.

@cmars, @wallyworld, will this PR affect the UX you expect?

Thank you all for trying out and improving MicroK8s.

[cmars]

@ktsakalozos I think this could possibly solve an open issue I've had with the wrong NIC getting chosen by microk8s, #92. Using localhost for everything sounds OK. I've been using microk8s exclusively in local VMs to get a more repeatable network environment as a result of that bug, which also mitigates exposure to the bind issue.

I tried installing 1.13/edge/security-testing on the host I have that's affected by #92, but it still doesn't work. Several issues I found:

1. microk8s.config still writes a kubeconfig with a non-localhost address in it. The IP didn't match the SAN IPs in the cert so I had to edit the kubeconfig and change the address to 127.0.0.1.
2. hostpath-provisioner and kube-dns are both stuck in CrashLoopBackoff. They're getting a timeout trying to connect through the cluster IP on cbr0, like this:

hostpath-provisioner:

F0103 00:15:43.787810       1 hostpath-provisioner.go:162] Error getting server version: Get https://10.152.183.1:443/version: dial tcp 10.152.183.1:443: i/o timeout

kube-dns:

I0103 00:19:08.529783       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
F0103 00:19:09.029801       1 dns.go:167] Timeout waiting for initialization

kube-dns sidecar:

W0103 00:22:33.761886       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:33805->127.0.0.1:53: read: connection refused

Tried installing a helm chart and also got a dial tcp 10.152.183.1:443: i/o timeout.

[gbevan]

@ktsakalozos

@gbevan I think the proposed patch does exactly what you describe. The API service is available on port 16443 over SSL and users have to present proper credentials.

I see in PR#88 it is refering to a static password file for 16443 access. Thats fine for user access, but what about app-2-app? I'd like to be accessing the api from devops/automation tools like jenkins etc.

[gbevan]

I'm using this gist approach for now - leaving iptables FORWARD rule as DROP and explicitly enabling https on 6443 to the proxy, blocking access to 8080...

[ktsakalozos]

I'd like to be accessing the api from devops/automation tools like jenkins

@gbevan you can edit /var/snap/microk8s/current/credentials/basic_auth.csv and add whatever passwords you want.

[sixcorners]

Binding to localhost is better but you are still vulnerable to dns rebinding attacks. Can this be a unix socket file like how docker does it? If not can it be secured using actual credentials?

[elwaxoro]

Note for people following behind that want the old behavior back: edit /var/snap/microk8s/current/args/kube-apiserver, set --insecure-bind-address=0.0.0.0, then restart with microk8s.stop and microk8s.start

[ktsakalozos]

Closing this since insecure port was bound to 127.0.0.1 for a long time and is now completely removed.

[sixcorners]

Isn't there something more that can be done to prevent against dns rebinding attacks?