search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
264 stars 59 forks source link

GlusterFS #236

Open schmitch opened 5 years ago

schmitch commented 5 years ago

Hello, currently I'm toying around with Fedora CoreOS after running Container Linux in production for many years now. What I wonder is, will it be easier to get GlusterFS mounts running on Fedora CoreOS? It was always a pain in Container Linux.

(I'm relativly new to rpm-ostree, but familiar with rpm/dnf/yum)

schmitch commented 5 years ago

ah found it:

# rpm-ostree install glusterfs

this is really cool.

bgilbert commented 5 years ago

We discourage the use of rpm-ostree package overlays on Fedora CoreOS, since they can complicate automatic upgrades. We should decide whether to ship GlusterFS directly in the OS.

schmitch commented 5 years ago

I've actually tried to run kubespray on fedora coreos and I'm not sure if it is possible to run it without any rpm-ostree package overlays. The first two packages which kubespray needed were:

at the moment it runs like a "Container Linux", but these two are needed for distributions without python. I still couldn't get the libselinux-python bindings to work, but I guess I just need some package that the interpreter will work.

So I basically just played an hour or so with it and found the rpm-ostree install really useful. Especially when working with ansible, it's really helpful.

smekkley commented 5 years ago

@schmitch CoreOS runs kubelet in rkt. In case of FCOS, it's possible to do the same with podman. The official kubelet docker image has gluster client as well.

schmitch commented 5 years ago

I'm not sure but I do not thing that k8s encourages to run kubelet inside a container. At least kubeadm, kops and kubespray discourage it.

(From Kubespray):

* *kubelet_deployment_type* - Controls which platform to deploy kubelet on.
  Available options are ``host`` and ``docker``. ``docker`` mode
  is unlikely to work on newer releases. Starting with Kubernetes v1.7
  series, this now defaults to ``host``. Before v1.7, the default was Docker.
  This is because of cgroup [issues](https://github.com/kubernetes/kubernetes/issues/43704).

It's not impossible, just not encouraged

smekkley commented 5 years ago

I wonder if kubernetes project actually discourages it as the kubelet docker image come from the project itself. Also because solutions like typhoon just does it.

I wouldn't run system containers on docker either because rkt / podman have more granular options and is meant to be demonized through systemd like a normal process.

bgilbert commented 5 years ago

Containerizing the kubelet was deprecated in https://github.com/kubernetes/kubernetes/pull/74267 and support was removed in https://github.com/kubernetes/kubernetes/pull/80043.

smekkley commented 5 years ago

I think it's talking about containerized flag. Running kubelet in rkt doesn't need behaviour change from kubelet because you are supposed to run it without network and pid isolation and bind mount the most of the local filesystem.

schmitch commented 5 years ago

actually I think it would make more sense to not containerize kubelet and provide a way to install stuff via rpm-ostree install glusterfs I mean if it would be possible to limit the scope of the packages it probably wouldn't disturb the upgrade process that much.

Basically most packages that are useful would be:

I do not think that "much" more is needed. I mean running kubespray will run onto the problem that the bootstrap python needs libselinux and thus stuff like python3-libselinux would be needed, but maybe there are some workarounds for that. also the libraries for bootstrapping would be needed with kubespray anyway.

smekkley commented 5 years ago

Probably for Kubespray, it's impossible without direct support from the OS. I guess this issue is also dependent on #93 and #37 .

I'm really curious and looking forward to decision on this. It really affects everything like OS size and how FCOS is provisioned.

I for one prefer everything containerized solution like CoreOS, not only kubelet, but also things like etcd. But I also understand it's useful to have native support. For some people it just looks like unnecessary layer to have a container, and it is. But for me it's more about having less dependencies on OS.

smekkley commented 5 years ago

@bgilbert It's indeed only --containerized option that's deprecated. https://github.com/kubernetes/kubernetes/issues/74148#issuecomment-465323588

schmitch commented 5 years ago

@smekkley actually from the issues running kubelet inside a container is neither tested nor supported. and the ability might go away completly in the future. the issues where the flag was removed also discusses this. So basically I still think it's important to make it easy to add "some" packages to the base install, like filesystem drivers. glusterfs was always a huge pain under CoreOS.

smekkley commented 5 years ago

My point was that kubernetes can't remove support as long as it supports running natively. So the ability cannot go away. They need to stop creating kubelet docker image if they want to stop supporting. If you use CoreOS, it's pretty obvious. Nobody needs --containerized option.