Open EBS CSI driver

[Jean-Baptiste-Lasselle]

Then finally, I can try OpenEBS CSI driver, as mentioned here : https://github.com/3forges/packman/issues/12#issuecomment-2127925050

Note about disk space, I can add a 120 GB disk for just like 8 euros per months more, that will do 20 euros per months not more:

[Jean-Baptiste-Lasselle]

I had to change region to get an ARM64 t2a family VM (t2d are actually x84_64 amd64 ):

• t2A are only available in 3 regions in the world, us-central1 in Iowa , or netherlands, or in asia
• the cheapest was in iowa us-central1, so i swithc all resources to that region, and i got this new pricing calculator forecast:

oh, but i just got:

• fixed: there was not any availability of 't2a-standard-4' in zone us-central1-c (the erreor above is in us-central1-c), but it worked in zone us-central1-a.

https://www.googlecloudcommunity.com/gc/Infrastructure-Compute-Storage/Where-are-T2A-instances-available/m-p/611301

[Jean-Baptiste-Lasselle]

next error to solve :

[Jean-Baptiste-Lasselle]

[Jean-Baptiste-Lasselle]

ok, open ebs is either a local storage based provider, or it can have something else which ahs replicaton capabilities: mayastor.

The only problem with mayastor is that it requires working on x86_64, plus it is aimed towards ssdd (which is very expensive, even if the goal there is to get performance for storage:

So I'd say I prefer trying out first Rook, befoe going any further with open ebs

[Jean-Baptiste-Lasselle]

also note: my open ebs deployment still has a few isues, etcd and loki seem to have permissions issues with their mounted volumes:

oh and i found what's wrong with etcd , it is not arm64, it is x86_64 or amd64, but not arm:

and confirmed, ther is no arm64 docker image available:

so ther i would have to build a custom one.

if we had to stay on local storage, because it seems to be kind of well working, then we would have to work ourselves on a replication feature, that is another good reason why trying rook already in the next try

[Jean-Baptiste-Lasselle]

Ok, i have more about this issue with etcd and loki :

• first, i tested disabling mayastor and the OpenEBS csi driver works like a charm for jupyterhub
• yet: the only csi driver that i can use in this mode is the local-hostpath which storage class name is openebs-hostpath: well there not much of a mind blowing storage solution there
• so i would like to find out if it is possible to ry and use the LVM related option in OpenEBS:
  • first, i wil do that with mayastor still disabled, and applying the interesting workaround from here : https://github.com/protosam/tutorial-rook-ceph-in-kind
  • when I am done with that one first step, i'll see this: ok from there i will indeed have the possibility to try the Rook Ceph operator, because it will then have LVM volume group on the cluster nodes. Yet, I am not sure it wil work as the https://github.com/protosam/tutorial-rook-ceph-in-kind tutorial says "oh you just have to create LVM volume groups on linux", but i do'nt see how rook ceph can run LVM commands like creating logical volumes for each pvc (there i can create custom kind cluster node docker images and i just install lvm2 inside, we'll see)
• I could check by disabling mayastor, that both the etcd and loki containers that are problematically provisioned only for amd64, are pulled in only by mayastor, actually by dependencies of the mayastor chart
• well okay then, now there is still the question: can i get replication with mayastor ? Honestly, i don't feel it well, because mayastor is targetting x86_64 intel processors, but yet:
  • after i fully POCed the k8sup backup restore liefcycle,
  • then i can do one thing i found out: it is indeed possible to override the values of a helm chart that is a dependency of another helm chart. Yet here i have a second order dependency : openebs chart > mayastor chart > bitnami/etcd chart (and openebs chart > mayastor chart > whatever/loki-stack chart. SO i'll se if i can override valuess of second order depenecies with helm charts, we'll see. Note the helm cahrt for mayastor is in a repo named mayastor-extensions

Alright, the road is long to have a proper kubernetes storage architecture, isn't it ?

the properties i would want for that storage:

• backup / restore :
  • of jupyterhub users workspaces to an external minio : that minio will be in its own separate k8s cluster with its own disks, eventually. plus the users have git to permanently save their notebooks, and version conrol them btw
  • of postgres, mongo, and duckbd : perhaps not, data will be much too big , no ?
  • and what about lakefs, which has its storage into minio?I dont want to backup a minio in another minio, instead i should get a solid replication enabled minio
• okay so i really need a solid replication enabled minio: idealy across clusters using liqo, that way the data may be big, but still the resilience is high, and i need to assess k8s ability to do incremental backups?
• I want postgress, duckdb and mongo to use label, affinity anti affinity for them to use different cluster nodes and if they are doomed to use a local hostpath storage, well that at least that storage is dedicated to one db
• i want the terraform to preserve disk data (terraform destroy the vm, i keep the disks as is and when i re terraform all, the disks get re-attched etc....

ok thre's big work to get disaster recovery plan, with optimized performance/costs,

[Jean-Baptiste-Lasselle]

I determined that OpenEBS is really mean: if you don't use mayastor, their LVM provisioner is really stupid, it is not able to provision LVM volumes from multiple / kubernetes cluster node, you are boundto only one node. That's why I switchedhappily to TopoLVM