search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.32k stars 926 forks source link

Support for Kubernetes / Mesos #670

Closed monokal closed 5 years ago

monokal commented 9 years ago

I know this isn't really the place, it's also been raised on the Kubernetes side, but if anyone's willing to put in some hours to kick this off that'd be great.

Kubernetes recently implemented container runtime abstraction (well, mainly the CoreOS guys to allow for rkt integration - which is now fully implemented) to allow for more than the initial Docker support. This means it's now possible (and relatively simple I believe after chatting with the K8s guys) to add in LXD support, which would be...spectacular.

Having dug through the repo, it would seem that we would need something similar to what's in the "dockertools" directory here (also note the rkt directory): https://github.com/GoogleCloudPlatform/kubernetes/tree/master/pkg/kubelet

If anyone has any time, or even just suggestions, shoot away (I appreciate there's higher priorities in LXD itself at the moment).

stgraber commented 9 years ago

Hi,

Kubernetes support for LXD would be pretty nice though indeed it's not on our current roadmap for the next 6 months or so. Of course this doesn't prevent you or somebody else from doing it and we would certainly be happy to help review/test anything you come up with.

Thanks!

kapilt commented 7 years ago

kubernetes issue is https://github.com/kubernetes/kubernetes/issues/6862

techtonik commented 7 years ago

And Vagrant support would be nice as well. And some sponsoring to speed up the process.

candlerb commented 7 years ago

As for Mesos, it looks like there are two different approaches which could be taken.

  1. The docker containerizer delegates to docker daemon via the docker CLI. So a new lxd containerizer could be written which calls out to the lxd CLI or API.

  2. The mesos containerizer (youtube) has plugins for "provisioners" (image fetch/setup), "isolators" and "launchers". It is now capable of running docker containers directly without using docker daemon.

    A similar approach could be taken for lxd containers. The containers would then be able to run without using the lxd daemon (although they would not be visible to lxc list etc)

techtonik commented 7 years ago

https://kubernetes.io/docs/getting-started-guides/ubuntu/local/

stgraber commented 7 years ago

@techtonik that's unrelated. This issue is about supporting LXD as the runtime of Kubernetes or Mesos, not about running Kubernetes or Mesos on top of LXD which is indeed possible today.

techtonik commented 7 years ago

What do you mean by "supporting LXD as runtime of Kubernetes"? Is it that LXD containers are running Docker containers inside?

monokal commented 7 years ago

k8s_in_lxd != lxd_in_k8s

On 3 May 2017 1:57 a.m., "anatoly techtonik" notifications@github.com wrote:

What do you mean by "supporting LXD as runtime of Kubernetes"? Is it that LXD containers are running Docker containers inside?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lxc/lxd/issues/670#issuecomment-298801149, or mute the thread https://github.com/notifications/unsubscribe-auth/AE6NksWBjf-yHGqizipl9YYTOEfOHzEhks5r19EOgaJpZM4EdLR9 .

techtonik commented 7 years ago

@monokal k8s is always on top of Docker, and I can't see how Docker can be on top of k8s. k8s master can be run on plain OS, so I don't understand what else is needed from it to manage LXD instances https://github.com/kubernetes/kubernetes.github.io/issues/3566#issuecomment-297906385

monokal commented 7 years ago

You've either misunderstood the topic of this thread, or the content of the link you posted.

This thread is discussing the implementation of LXD container runtime support for Kubernetes/Mesos, so LXD containers can be managed within the cluster in the same way Docker/RKT runtimes are managed now.

Your link is related to running Kubernetes itself inside of LXD, for local development or whatever.

On 3 May 2017 2:07 a.m., "anatoly techtonik" notifications@github.com wrote:

@monokal https://github.com/monokal k8s is always on top of Docker, and I can't see how Docker can be on top of k8s. k8s master can be run on plain OS, so I don't understand what else is needed from it to manage LXD instances kubernetes/kubernetes.github.io#3566 (comment) https://github.com/kubernetes/kubernetes.github.io/issues/3566#issuecomment-297906385

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/lxc/lxd/issues/670#issuecomment-298802258, or mute the thread https://github.com/notifications/unsubscribe-auth/AE6NkvVZfoJwmpL371OfiZilY4uRuJKQks5r19NggaJpZM4EdLR9 .

techtonik commented 7 years ago

Your link is related to running Kubernetes itself inside of LXD, for local development or whatever.

What do you mean by "Kubernetes" here? If Master Node is running inside LXD and is managing other LXD containers, everything is fine. The page I referenced doesn't mention that Kubernetes is not managing LXD containers, but Docker containers that are running inside LXD containers.

monokal commented 7 years ago

"... and is managing other LXD containers... ". It's not. As it stands, Kubernetes does NOT support running LXD containers. All you're doing here is wrapping Kubernetes, Docker and everything else up in an LXD container. It doesn't give Kubernetes any ability to manage LXD, it's managing Docker inside the LXD container.

On 3 May 2017 2:22 a.m., "anatoly techtonik" notifications@github.com wrote:

Your link is related to running Kubernetes itself inside of LXD, for local development or whatever.

What do you mean by "Kubernetes" here? If Master Node is running inside LXD and is managing other LXD containers, everything is fine. The page I referenced doesn't mention that Kubernetes is not LXD containers, but Docker containers that are running inside LXD containers.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lxc/lxd/issues/670#issuecomment-298803809, or mute the thread https://github.com/notifications/unsubscribe-auth/AE6NkqBd9E1c7YX4vvpwHqs6gWudhynKks5r19bugaJpZM4EdLR9 .

techtonik commented 7 years ago

I see. Then https://github.com/kubernetes/community/blob/master/contributors/design-proposals/container-runtime-interface-v1.md and https://github.com/kubernetes/kubernetes/issues/28789 are needed to track this issue.

monokal commented 7 years ago

CRI is what we were waiting for (it wasn't implemented at the time this thread was opened), now all we need is someone with the time to wire it up to the LXD REST API (https://github.com/lxc/lxd/blob/master/doc/rest-api.md ).

On 3 May 2017 at 02:32, anatoly techtonik notifications@github.com wrote:

I see. Then https://github.com/kubernetes/community/blob/master/ contributors/design-proposals/container-runtime-interface-v1.md and kubernetes/kubernetes#28789 https://github.com/kubernetes/kubernetes/issues/28789 are needed to track this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lxc/lxd/issues/670#issuecomment-298804787, or mute the thread https://github.com/notifications/unsubscribe-auth/AE6NkvjM74TK-7iuSsig9FrhKfdfm9U0ks5r19kkgaJpZM4EdLR9 .

techtonik commented 7 years ago

I would be interested to do this, but there are no employers who back up me.

techtonik commented 7 years ago

I've fixed Kubernetes docs to avoid future confusion - https://kubernetes.io/docs/getting-started-guides/ubuntu/local/

jseguillon commented 7 years ago

Does anyone know how much effort this would be to implement CRI - LXD ? Maybe we could do this all together if no one founds a company back up?

techtonik commented 7 years ago

On Thu, May 11, 2017 at 10:55 PM, jseguillon notifications@github.com wrote:

Does anyone know how much effort this would be to implement CRI - LXD ? Maybe we could do this all together if no one founds a company back up?

I can not do anything together with $5 in my pocket, so don't count on me.

techtonik commented 7 years ago

Watching https://www.youtube.com/watch?v=rHUngcGgzVM&feature=youtu.be&t=309 about Unified containerizer concept from Mesos. How much of the following from Unified containerizer is already implemented by LXD?

(it already looks like this ticket should be split)

enzian commented 7 years ago

I'm interested in that too and I'm ready to contribute - I've started writing a CRI shim for LXD which I can publish as soon as there is interest and I've made some more progress! This is not going to be an overnight thing and I'd need help doing this in the long haul. My company is currently backing this as a research project with 10% of my weekly work time.

I would keep the code in a dedicated repository which as soon as it gets some traction could be moved to under the LCX organizational wing!

So, who ever is interested in contributing to this project - please drop me a line here so we can coordinate!

brauner commented 7 years ago

Sorry, wrong button. :)

caiobegotti commented 7 years ago

Hey Michael, that sounds great! Have you pushed anything to the repo yet so we can take a look at it too and ser how we could contribute?

Cheers

On 22 Jun 2017, at 03:51, Michael Ingold notifications@github.com wrote:

I'm interested in that too and I'm ready to contribute - I've started writing a CRI shim for LXD which I can publish as soon as there is interest and I've made some more progress! This is not going to be an overnight thing and I'd need help doing this in the long haul. My company is currently backing this as a research project with 10% of my weekly work time.

I would keep the code in a dedicated repository which as soon as it gets some traction could be moved to under the LCX organizational wing!

So, who ever is interested in contributing to this project - please drop me a line here so we can coordinate!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

enzian commented 7 years ago

@caio1982: not yet but somewhere in the next two weeks - currently in a fight with golangs vendoring system...

enzian commented 7 years ago

Well - why delay this: I checked in the initial project structure which is loosely based on what the rktlet guys created. Suggestions, ideas and mostly help dearly welcomed! :-)

richil-bhalerao commented 7 years ago

Hey guys, Any update on this one? We were curious to know if this support is going to be provided sooner or later. Thanks!

shaneutt commented 7 years ago

I'm also interested in any updates on this issue, thank you!

enzian commented 7 years ago

Just as a heads-up for everybody looking for a solution that provides more hypervisor like isolation - you might be able to solve your problems with frakti. Which has the benefit of being a much more mature container runtime!

naototty commented 5 years ago

I found "lxe" project. https://github.com/automaticserver/lxe

https://discuss.kubernetes.io/t/lxe-released-a-kubernetes-integration-of-lxc-lxd/3022

stgraber commented 5 years ago

Given that LXE is a thing these days and this issue was there mostly for tracking/discussion purposes, I think it's time to close it.

That obviously doesn't mean we can't comment some more on there but I would suggest that https://discuss.linuxcontainers.org may be a better place to chat about Kubernetes/LXD integration.

gattytto commented 4 years ago

take a look at this, minikube (single node kube cluster) can be run inside a LXC container: https://github.com/eclipse/che-docs/pull/885/files