support bcachefs

[xmonader]

https://bcachefs.org/

[muhamadazmy]

There are a lot of thought needed to put into this for the following reason:

• Backward compatibility to support both is possible on a small scale. A node can either run the new style storage or the old style, since a zos node already formats and takes over all attached disks on start.
• this means that only new nodes can have bcachefs
• bcachefs is not on main stream kernel hence initramfs need to be also changed to add that support.
• bcachefs itself can be used pretty much like btrfs, but does not support quota on subvolumes same way as btrfs does (as far as i understand) we relay heavily on quota to control the size of a VM (writable layer)
• Since all HDD will be part of the same bcachefs (i assume) then zdbs need to work against subvolumes not against full disks as per before, hence changing all the storage daemon logic on how disks are assigned, and used.

Other issues/concerns might show up while actually working on this. But in all cases that is a lot of changes, that gets more complex to be backward compatible as shown before (support running old and new style storage) and deciding if we gonna need to support both will change the amount of compatibility code we will have to drag.

This combined with mycelium work, i really start to think we need to have a new zos version 4 that runs separately from the current version 3

[muhamadazmy]

On another hand there is also a bcache (not bcachefs) that is similar to bcachefs but on the device level (not filesystem level) which means we can still use the devices with btrfs which can minimize the changes required to support bcaching

[xmonader]

Shouldn't be blocked anymore right?

[xmonader]

bcachefs is also now merged on trunk

[muhamadazmy]

My takes over using bcachefs or bcache in zos, and what i think might be a better solution. To make it clear, i will try to explain the different between both and why i think neither is good for our use case.

• bcache: uses one ssd device to cache read/write IO to slower hdd devices. Then you put your filesystem of choice on the bcache device.
• bcachefs: like the above, but instead it creates a filesystem instead. Hence it becomes the filesystem, and you can't for example use btrfs, or other filesystems.

bcachefs

Why I think bcachefs is the right choice, is the following:

• First of all (and the most obvious) zos storage depends heavily on btrfs, while this can be modified, but it requires a lot of work plus the incompatibilities between the 2 filesystems features.
• Second and most important is that we right now uses files as virtual disks that are attached to virtual machines. This itself is hurting the performance of the disks inside the vm very badly. sine an IO request has to go through many layers (fs opertaion (VM) -> IO to device (VM) -> this then goes over virtio to underlying file, which is then translate to fs operation (host) -> then io to block device (host)).
• If we wanna gain performance, it would be nicer if we replace this model with a more direct approach. For example, the vdisk is directly mapped to a device on the host system

bcache

To solve the above problem i would think that bcache will be a better option. Since we can create a bcache device witch using the ssd as a cache device, then we will can create paritions (for each vdisk) that then can be attached directly to the VM, which will make the performance way better not just because we eliminate the underlying layers, but also because of that we have ssd device used as a cache.

The problem with bcache imho is that we will need to create partitions. Which will cause fragmentation of the disk and loss of space (imagine deleting a disk then u have an unused space in the middle of the disk that u can use because it's smaller than the newly requested disk)

This can be improved by using something like LVM on top, but LVM is a very old technology and i read that btrfs has big issues with it.

My proposed solution

Instead of using any of the above technologies we can build our own virtual disk, with nbd. The implementation will use the SSD for the cache (hot blocks), and evict the least accessed blocks to HDD. The service will still use files, but with few tricks this can be optimized for speed.

We can do really nice things with this then, for example, we can write the back end to multiple HDDs, we can then do mirror, strip and even do erasure coding to make sure loss of one of the HDDs does not cause loss of the data.

We later can even use (instead of HDD) use remote, distributed ZDBs on near by machines. This solves many issues:

• Use SSD for read/write operations
• Use HDD (or any other store implementation) to store colder blocks
• No fragmentation, since the nbd server will just use files on the hdd, deleting an nbd device will just delete the packing files.
• Another good added value is that with this it means that we can actually have a single vdisk that spans multiple hdd and effectively use sizes that are bigger than a single HDD.
• Configure a backend pipeline that can be built to implement mirroring and erasure coding for more important workloads per user request.

[muhamadazmy]

I did some experiments with bcachefs and I was thinking what is the best way to actually use this with ZOS given the following:

• ZOS is completely autonomous, in a sense that a farmer has no access to it, nor any management tools. A failure of a disk might not be clear to the farmer to replace. There is no way to communicate what disk needs replacing
• A zos node can have a lot of ssd and hdd devices, it makes sense to leave some of them unused until they are needed which means not all disks can be part of bcachefs from the start.
• If we decided to create a single bcachefs filesystem that includes all ssd/hdd, this would work the best in terms of performance and space management (specially with replication) BUT at the same time we can get into issues because of point number one. What to do if a disk is dead? how to tell the farmer which disk to replace, and how to restore the setup.
• Another approach is to create a ssd part + hdd part for each user subvolume, this is better in terms of controlling quota and the ssd vs hdd usage. It also mean we can serve users different storage policies (replication, ssd to hdd percentage, etc...) but at the same time we can have fragmentation issues (when free spaces are scattered a cross a single disk)
  • This can be improved by setting some limitations on the sizes of partitions (say they has to be multiple of 10G always) then we can create all possible partitions at once, then allocate partitions (or more) per user workload.
  • A single ssd+(parts) can be then formatted and mounted, and attached to user workloads
  • A loss of a device can cause workloads on that device to get lost given there is no replication enabled for that workload
• If we use a single bcachefs system on all devices, while we won't have fragmentation issue, we will still hit an issue with, loss of disks and controlling quota per subvolume

[xmonader]

due to blockers, we won't address in 3.14