Rook Integration

[andrewrynhard]

Problem Description

Users want an easy way to enable storage in their clusters.

Solution

The high level overview is:

• Focus on RBD (S3 and CephFS will come later)
• Requires at least 4 nodes
• Each storage node will need a set of kernel args
• Each storage node requires at least 2 disks
• Deployed using Rook manifests with recommended settings
• Upgrades will come in phase 2 of the development of this feature
• User configurability is not part of this initial release (customization will come later)

The idea is that when a user requests storage we can filter machines and require specific things that we know are needed in order to deploy Rook with production quality settings.

The biggest question is how do we want to manage the manifests? We could use

1. Helm
2. Kustomize
3. Custom manifests management solution

Alternative Solutions

No response

Notes

No response

[ro11net]

I love the idea of an easy route to deploy storage on an Omni cluster! Have you guys thought about using Longhorn to accomplish this now that it's compatible with Talos as of Longhorn 1.6.0?

While I like rook-ceph more for large clusters, Longhorn is what I'm using right now on Omni due to how lightweight and easy to deploy it is and it works regardless of the number of nodes or drives in the cluster.

[andrewrynhard]

Hey @ro11net. Great to see you here! Right now we are focusing on Rook because most of the enterprises we are working with use Ceph. Over time we may consider other CSIs but the depth of integration we have planned with Ceph is something that would be hard to replicate for every CSI out there. We aren't looking at just installing it but orchestrating upgrades, validating data with APIs, etc. So the storage provider would need to provide similar APIs.

[yoyellow]

Got any love planned for external ceph clusters? Rook can be used for this but is overkill obviously..