How install a hetzner volume and share data across pods?

[cryptobench]

I read the README and as far as I understood, you do some magic to share the volume between all pods. Is that correctly understood?

I am having some trouble provisioning this, and this is the PVC i've tried creating

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mediafiles
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: hcloud-volumes

And in my deployment:

...
    spec:
      volumes:
        - name: media-storage
          persistentVolumeClaim:
            claimName: mediafiles
      restartPolicy: Always
      containers:
      - name: stats-backend
        volumeMounts:
          - mountPath: "/data"
            name: media-storage
...

But my container stays in a creating state, so I must be doing something wrong. Any clue to what i'm doing wrong?

[vitobotta]

Uhm the code seems correct. Is the volume in bound state or not?

BTW Hetzner storage is block storage, so a volume can only be attached to a single pod per time. How many pods is your deployment running?

[cryptobench]

Ah I see. I understood the README as if it allowed to be attached to multiple pods at the same time. Since that's not the case, then I won't use it. I need to distribute some files across all multiple pods, so I hoped it would solve that issue for me.

Again, thank you very much for the time you put into this project! Already got more bang of my buck + performance with this setup.

[vitobotta]

Regular block storage, regardless of the Kubernetes cluster and storage solution, can only be attached to a single pod per time. If you want multiple pods accessing the same storage you can use something like the NFS Server Provider or OpenEBS NFS provider (https://github.com/openebs/dynamic-nfs-provisioner).

[vitobotta]

So that I can improve the README if needed, can you please tell me which section seemed to suggest that you can attach the same volume to multiple pods? Thanks

[cryptobench]

So that I can improve the README if needed, can you please tell me which section seemed to suggest that you can attach the same volume to multiple pods? Thanks

Yes sure, it's this part:

Once the cluster is ready you can create persistent volumes out of the box with the default storage class hcloud-volumes, since the Hetzner CSI driver is installed automatically. This will use Hetzner's block storage (based on Ceph so it's replicated and highly available) for your persistent volumes. Note that the minimum size of a volume is 10Gi. If you specify a smaller size for a volume, the volume will be created with a capacity of 10Gi anyway.

Specifically replicated "This will use Hetzner's block storage (based on Ceph so it's replicated and highly available)"

[cryptobench]

Regular block storage, regardless of the Kubernetes cluster and storage solution, can only be attached to a single pod per time. If you want multiple pods accessing the same storage you can use something like the NFS Server Provider or OpenEBS NFS provider (https://github.com/openebs/dynamic-nfs-provisioner).

Thank you very much for the suggestion! I will take a look at it!

[vitobotta]

So that I can improve the README if needed, can you please tell me which section seemed to suggest that you can attach the same volume to multiple pods? Thanks

Yes sure, it's this part:

Once the cluster is ready you can create persistent volumes out of the box with the default storage class hcloud-volumes, since the Hetzner CSI driver is installed automatically. This will use Hetzner's block storage (based on Ceph so it's replicated and highly available) for your persistent volumes. Note that the minimum size of a volume is 10Gi. If you specify a smaller size for a volume, the volume will be created with a capacity of 10Gi anyway.

Specifically replicated "This will use Hetzner's block storage (based on Ceph so it's replicated and highly available)"

That, as it says, means that the storage itself is replicated by Hetzner, it doesn't say that it can be attached to multiple pods :) This is not a limitation with Hetzner's block storage. Any block storage with any provider can only be attached to a single pod per time; due to how block storage works, multiple pods accessing the same volume concurrently would lead to data corruption. The same thing applies to clusters you create in GCP, AWS, Azure, DigitalOcean - basically any block storage.

You can have multiple pods access the same data, but with specific filesystems that allow it, not block storage. I mentioned NFS because it's something easy that you can set up in any cluster, but other examples are the distributed file systems offered by GCP, AWS etc. Ceph also has such a file system, and to use that you would need to set up Rook in Kubernetes. The NFS option with e.g. OpenEBS is perhaps the easiest.

[aleksasiriski]

Uhm the code seems correct. Is the volume in bound state or not?

BTW Hetzner storage is block storage, so a volume can only be attached to a single pod per time. How many pods is your deployment running?

Actually any ReadWriteOnce volume can be attached to multiple pods, but only on a single node. So you can set pod affinity to require being on the same node for desired pods and share that volume across the pods!

[cryptobench]

Uhm the code seems correct. Is the volume in bound state or not? BTW Hetzner storage is block storage, so a volume can only be attached to a single pod per time. How many pods is your deployment running?

Actually any ReadWriteOnce volume can be attached to multiple pods, but only on a single node. So you can set pod affinity to require being on the same node for desired pods and share that volume across the pods!

I am new to Kubernetes, but i'm wondering if this really benefits one in any kind of way? My main goal is having as high availability as possible, and requiring pods to be on the same node to use the storage in a shared manner, doesn't seem like it would benefit in any way.

[vitobotta]

Uhm the code seems correct. Is the volume in bound state or not? BTW Hetzner storage is block storage, so a volume can only be attached to a single pod per time. How many pods is your deployment running?

Actually any ReadWriteOnce volume can be attached to multiple pods, but only on a single node. So you can set pod affinity to require being on the same node for desired pods and share that volume across the pods!

You are totally right, my bad. I forgot that pods on the same node can share the volume :)

[vitobotta]

Uhm the code seems correct. Is the volume in bound state or not? BTW Hetzner storage is block storage, so a volume can only be attached to a single pod per time. How many pods is your deployment running?

Actually any ReadWriteOnce volume can be attached to multiple pods, but only on a single node. So you can set pod affinity to require being on the same node for desired pods and share that volume across the pods!

I am new to Kubernetes, but i'm wondering if this really benefits one in any kind of way? My main goal is having as high availability as possible, and requiring pods to be on the same node to use the storage in a shared manner, doesn't seem like it would benefit in any way.

If high availability is a requirement then one option would be to set up Rook and set up a Ceph filesystem. I have used Rook for block storage in the past, not filesystems so I can't tell much about how good it is, but it's Ceph so it should be pretty solid. https://rook.io/

[aleksasiriski]

Uhm the code seems correct. Is the volume in bound state or not? BTW Hetzner storage is block storage, so a volume can only be attached to a single pod per time. How many pods is your deployment running?

Actually any ReadWriteOnce volume can be attached to multiple pods, but only on a single node. So you can set pod affinity to require being on the same node for desired pods and share that volume across the pods!

I am new to Kubernetes, but i'm wondering if this really benefits one in any kind of way? My main goal is having as high availability as possible, and requiring pods to be on the same node to use the storage in a shared manner, doesn't seem like it would benefit in any way.

I'm also new to kubernetes, you learn A LOT everyday (: To achieve true HA your apps NEED to have external database support (postgresql and/or redis for caching).

Another solution would be to use ceph/glusterfs/nfs and have ReadWriteMany volumes but trust me it's not worth it if that volume stores SQLite DB, it's better to ask the devs to implement postgres...

If you need shared files like assets other than SQLite then you can use Longhorn RWX volumes or Rook Ceph hosted on that same cluster since Hetzner (and a lot of other providers) don't support RWX yet.

[cryptobench]

I am currently using DigitalOceans S3 service named Spaces, but there's quiet a big performance hit when using it to sharing files across pods as I first have to download them, then upload them, then on the other pod download again and so on... so i'm definitely looking for that sweet shared storage system to use with K3/K8's

[vitobotta]

Uhm the code seems correct. Is the volume in bound state or not? BTW Hetzner storage is block storage, so a volume can only be attached to a single pod per time. How many pods is your deployment running?

Actually any ReadWriteOnce volume can be attached to multiple pods, but only on a single node. So you can set pod affinity to require being on the same node for desired pods and share that volume across the pods!

I am new to Kubernetes, but i'm wondering if this really benefits one in any kind of way? My main goal is having as high availability as possible, and requiring pods to be on the same node to use the storage in a shared manner, doesn't seem like it would benefit in any way.

I'm also new to kubernetes, you learn A LOT everyday (: To achieve true HA your apps NEED to have external database support (postgresql and/or redis for caching).

Another solution would be to use ceph/glusterfs/nfs and have ReadWriteMany volumes but trust me it's not worth it if that volume stores SQLite DB, it's better to ask the devs to implement postgres...

If you need shared files like assets other than SQLite then you can use Longhorn RWX volumes or Rook Ceph hosted on that same cluster since Hetzner (and a lot of other providers) don't support RWX yet.

Longhorn's RWX is still NFS, and I don't recommend NFS if high availability is a requirement anyway, unless you have some nice (and expensive) NetApp setup :D

[vitobotta]

I am currently using DigitalOceans S3 service named Spaces, but there's quiet a big performance hit when using it to sharing files across pods as I first have to download them, then upload them, then on the other pod download again and so on... so i'm definitely looking for that sweet shared storage system to use with K3/K8's

I may have another solution for you which can work great in many cases: Hetzner's Storage Box. It's a storage product (https://www.hetzner.com/storage/storage-box?country=fi) that allows you to mount the storage via CIFS, which is easy to set up in Kubernetes. I have used this setup even with Plex for media streaming and it worked well :D

This storage is cheap but is reliable (it's based on Ceph and replicated). If you are interested to try this I can give you hints on how to configure CIFS access.

[aleksasiriski]

I'm using Hetzner Storage Box as well, literally the cheapest HDD storage in the world. I used this driver to make it work: https://github.com/kubernetes-csi/csi-driver-smb

[cryptobench]

What's the performance like on the storage box? I have tons of read/write operations that might happen a second.

Also extra question, is it correctly understood that I can use longhorn and attach it to a Hetzner cloud volume and use RWX? I was looking at benchmarks and its around 600 MiB/s.

[aleksasiriski]

What's the performance like on the storage box? I have tons of read/write operations that might happen a second.

Hetzner Storage Box is limited because it's on spinning hard drives and also Samba/CIFS is slower than NFS. Longhorn is easiest to setup but Rook Ceph is more reliable.

Also extra question, is it correctly understood that I can use longhorn and attach it to a Hetzner cloud volume and use RWX? I was looking at benchmarks and its around 600 MiB/s.

Yes, you can. Also you can buy a cheap VPS just for longhorn and use it's NVMe storage as it's faster than Volumes, but that's not HA.

[fliespl]

@aleksasiriski do you have an example of rook ceph setup for hetzner-h3s? I.e. utilizing hetzner volumes attached to 3 worker nodes?

[aleksasiriski]

I never ended up using Rook, but Longhorn is supported to run using hcloud-volumes in kube-hetzner project if I remember correctly. Anyways, it should be easy to setup your self using their Helm chart and just mounting volumes manually/terraform and mounting in fstab.