Comparison and/or top level overview of the drbd-on-k8s ecosystem

kvaps / kube-linstor

Containerized LINSTOR SDS for Kubernetes, ready for production use.

Apache License 2.0

130 stars 25 forks source link

Comparison and/or top level overview of the drbd-on-k8s ecosystem #32

Closed t3hmrman closed 3 years ago

t3hmrman commented 3 years ago

Thanks for all your work on this project!

I've been recently doing some evaluations on the storage plugin ecosystem for k8s and came across LINSTOR, this project, and piraeus plugins that use and extend drbd9. I'd love some documentation on the differences in approach/implementation between these -- a small section in the README would be great.

Personally I'm just getting up to speed on how drbd9 works and what the options do and how well it works with Kubernetes -- up until now I've only looked at Ceph (via Rook), Longhorn (via OpenEBS Jiva), OpenEBS cStor and OpenEBS Mayastor. LINSTOR looks like an excellent alternative, but I'm a bit unclear as to the difference between the three project (this project, LINBIT/linstor-server and piraeusdatastore/piraeus-operator)

kvaps commented 3 years ago

Hi. That's not a bad idea. Will do.

Actually kube-linstor (this project) started earlier than the official piraeus-operator. Both projects have the same goals to containerize linstor-server and other components and provide ready solution for running them in Kubernetes. But they are doing that different ways, eg:

kube-linstor - is just standard Helm-chart, it consists of static YAML-manifests, very simple, but it has no some functionality inherent in piraeus-operator, eg. autoconfiguration of storage-pools (work in progress) and auto-injecting DRBD kernel-module.
piraeus-operator - implements the operator pattern and allows to bootstrap LINSTOR cluster and configure it using Kubernetes Custom Resources.

No matter what you choose, in both cases you will have the same components but installed the different way. Thanks to the LINSTOR logic, when you have separated control-plane and data-plane, you can try both and easily swap between the solutions very quickly.

t3hmrman commented 3 years ago

Thank you -- this explains a lot. I did not realize that LINSTOR server was below both of these projects -- I thought it was deployable in and of itself with CRDs (I haven't gotten to deploying this plugin just yet).

I'm currently evaluating all of these solutions against each other, excited to give them a try. I'm a bit torn because kube-linstor is lower complexity than piraeus-operator since I am indeed a fan of smaller, more straight forward tooling. I might actually even prefer static pool allocation since I'm dealing with a partition and a full disk (not just a number of fully empty disks). On the other hand, the operator pattern is really nice when it works properly (and IMO Kubernetes is actually nothing but a huge operator, unfortunately they didn't really realize this until later and CoreOS had to bring it up for people to realize it more broadly).

I guess I have to try both to get a feel for it, thanks for clarifying where both projects stand!

kvaps commented 3 years ago

Actually piraeus-operator is not difficult, but it holds some program logic inside :)

t3hmrman commented 3 years ago

Thanks for explaining -- also thanks for the presentation you gave, was a very informative watch!

@kvaps By the way, if you'd like me to add what you've said to the documentation and make a PR I would love to.

t3hmrman commented 3 years ago

Closing this since anyone who is looking for this information will be able to find it here.

kvaps commented 3 years ago

I think it's not a bad idea to explain users what the difference. So if you'd like to prepare PR, I'm going to merge it.