[Feature Request] Support microk8s in a docker image.

kkimdev commented 5 years ago

(updated to make a proper feature request)

It would be really great if we can use microk8s for CI. Since it's by far the fastest local Kubernetes solution, I think there will be a great value in that space. I was using https://github.com/kubernetes-sigs/kubeadm-dind-cluster for my project, but that adds ~10min setup time to each CI build.

For that, it microk8s should be able to run inside a Docker image, though I'm not sure what it takes to get there. Few issues I can see:

Performing Snap install
Running systemd
- https://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/
- https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/

marcoceppi commented 5 years ago

Snaps in docker are possible I'd try that route and report back!

kkimdev commented 5 years ago

@marcoceppi Oh nice, thank you! It would be really useful if I can run microk8s as a docker, and I believe that other people will find useful too!

FYI, There is a couple of projects https://github.com/kubernetes-sigs/kind https://github.com/kubernetes-sigs/kubeadm-dind-cluster can be used to build Kubernetes running in docker, but it's not fast, less mature yet, and then also I had an issue with kubeadm-dind-cluster recently so had to disable. If possible, I think microk8s can be a preferred alternative in that space.

kkimdev commented 5 years ago

Now I think about it, for the snap issue, maybe a better way is having a way to install and run microk8s without snap. Thr main benefit of Snap is sandboxing, but we're disabling on Docker anyways(from the link you provided). Also, not all environment has Snap support like Alpine linux or Windows wsl.

Edit: But I don't know how complex is to run without Snap, so not sure it will be worth or not.

ktsakalozos commented 5 years ago

Hi @kkimdev ,

+1 to using MicroK8s on the CI for the reasons you mention (light, fast install, isolation).

I can see how we could implement this feature. At the time we release the snap to the store we also create a docker image and push it to a repository. With proper tagging we could have matching docker images versions and snaps.

@kkimdev, I would like your help in making a stronger case for this feature. How would you use the MicroK8s docker image? The counter argument to your request is: how is the isolation offered by a docker image better than the isolation of a snap? Snaps are using the same linux kernel features as docker (namespaces, cgroups). As soon as you sudo snap remove microk8s MicroK8s is completely removed, nothing is left behind. Why should we put an extra layer around MicroK8s, but most importantly what stops you from using MicroK8s without the docker wrapping?

For reference,

we deploy and test MicroK8s on travis on every PR. For this we have to set --conntrack-max-per-core=0 for kube-proxy. See https://github.com/ubuntu/microk8s/blob/master/tests/patch-kube-proxy.sh
I am aware of https://github.com/spring-cloud/spring-cloud-kubernetes that run their tests on CircleCI with https://github.com/spring-cloud/spring-cloud-kubernetes/tree/master/spring-cloud-kubernetes-integration-tests

Thank you for using MicroK8s.

kkimdev commented 5 years ago

@ktsakalozos Yay, that's exciting ! =D

Many open source CI solutions are very Docker centric. Often, running everything on Docker images is a standard, preferred method if not it's the only way. Examples:

Drone IO -This is what I'm using - runs everything on Docker images. So for example, building a Docker image inside a job is done by Docker-in-Docker(DIND) https://discourse.drone.io/t/build-and-test-docker-images-with-dind/1808 https://github.com/drone-plugins/drone-docker/blob/6207bb9d8518d1d988731d0ae89abaa33abded50/docker/Dockerfile#L3
Gitlab CI runner is also very Docker centric. Likewise, DIND is a recommended way of building a Docker image https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-in-docker-executor "Docker-in-Docker works well, and is the recommended configuration"

Edit: I just looked up the other non open source CI providers you mentioned:

Travis CI uses a virtual machine per build https://docs.travis-ci.com/user/reference/overview/#linux so no Dockerized microk8s needed (as we already know).
Circle CI provides both docker and virtual machine options https://circleci.com/docs/2.0/executor-types/ So there is a value of having Dockerized microk8s for Circle CI.

kkimdev commented 5 years ago

Perhaps a stronger usecase: to run microk8s on Kubernetes. So if CI runs on Kubernetes, or it's a Kubernetes native CI https://engineering.opsgenie.com/cloud-native-continuous-integration-and-delivery-tools-for-kubernetes-e6ea34d308c that launches Kubernetes pod for each job, then we need a Docker support. To my understanding that's one of the reasons why Kubernetes testing sig's project https://github.com/kubernetes-sigs/kind builds a Kubernetes Docker image.

jemc commented 5 years ago

I'm also looking for a lightweight way to run kubernetes integration tests in a CI system that only supports docker container build agents. I think microk8s would be an attractive solution if it were possible to run it in a docker container.

ktsakalozos commented 5 years ago

For reference some links related to the issue:

krichter722 commented 4 years ago

I took a look at the links posted by @ktsakalozos and spend some days on this. My summary: There're scripted workarounds that allow to run snaps inside a container, but they're not a containerized solution. I don't think this path will lead a real microk8s container.

Approaches like https://github.com/ogra1/snapd-docker are a practical idea, however it's hard to port it to up-to-date Ubuntu base images, systemd and snap versions and I don't think the time is well spent. The maintenance effort will always be enormous in comparison to a sheer image. I tried to come up with my own script similar to https://github.com/ogra1/snapd-docker, I however couldn't get systemd to run inside Docker using an up-to-date Ubuntu, however managed to make it work using CentOS 7. However, I then had a ton of other trouble because I needed to imitate expertise in systemd, snap and Docker. There's probably a huge impact on performance and resource requirements as well.

It'd be awesome if the snap devs could provide a container which allows to run and maybe even install snaps. It needs to be well maintained solution because it involves a lot of troublesome combinations of complex software, such as systemd. According to https://bugs.launchpad.net/snappy/+bug/1841327 there's no such plan, maybe you can knock on their door again.

Imo, the most promising and sustainable approach is to provide microk8s as container without involvement of snap for execution. I'd be interested in supporting the development, however I have no clue where to start.

jglick commented 4 years ago

I'm also looking for a lightweight way to run kubernetes integration tests in a CI system that only supports docker container build agents.

For what it is worth, I have been using Kind for this and it works well. (I generally use Microk8s for local development.)

nhoughto commented 4 years ago

We use another kind for this, not as fast as microK8s, but faster than the sigs kind

https://github.com/bsycorp/kind

trulede commented 4 years ago

I've developed a CI tool which runs on/in MicroK8s, uses the Kubernetes API and builds Containers with a tool called Kaniko (from Google).

Kaniko is the magic part, because of that I don't need Docker (or Docker in Docker) kind of solutions, so all the problems you get with GitLab/Travis and other things ... are gone.

PaulWBrassard commented 4 years ago

@trulede I'm interested to know more.

Also +1 for support microk8s in a docker image.

SemanticBeeng commented 4 years ago

@krichter722

couldn't get systemd to run inside Docker using an up-to-date Ubuntu

Mind having another try using x11docker https://github.com/mviereck/x11docker ?

canonical / microk8s

[Feature Request] Support microk8s in a docker image. #233