TheLinuxGuy
commented
1 week ago Interesting that you submitted this a few months ago because it feels like a Nostradamus prediction of bcachefs and what just happened. Kent Overstreet - the developer of bcachefs has been banned for at least 1 cycle of kernel releases looks like.
"I'd just like to interject for a moment: What you're referring to as bcachefs, is in fact, btrfs plus tiered storage."
This is my original text. I have posted a copy of this over at "btrfs" on GitHub for better visibility, and linked them here. All discussion should take place here.
Hello fellow nerds!
Please allow me to... ehm, interject into your daily workflow for half an hour to drop a message that concerns us all. As the title suggests, this long text is I believe both worth reading and has direct consequences for other tickets and how they're handled by the btrfs team. (especially: #610 - adding tiered storage feature)
I apologize for misusing the ticket system like this, but am hoping that centralizing the discussion here leaves the technical bug reports and feature requests free from discussions. I also did not want to spam the kernel mailing list, since tiered storage seems to be implementable by btrfs-progs. I furthermore believe that many btrfs team members read both places. Thank you for your patience.
1. What's happening in the filesystem world and why does it concern us (btrfs)?
The Linux community needs a good, stable, featureful next-gen file system. While other closed-source operating systems are pushing new featureful versions of their filesystems like ReFS on Windows and Apple File System (APFS) on MacOS, we are in danger of falling behind. I think with the current drama around Kent Overstreet and his bcachefs project, specifically his inability to work together with others as demonstrated on repeated occasions, it has become clear that btrfs is now the best (because: only) candidate for a next-gen filesystem on Linux.
We are the only ones that can finish the task in time.
Kent has severly aggravated both Linus Torvalds himself by repeatedly pushing thousands of lines of code at once in mid-release cycle (source), as well as people from the Debian project whom he severely pushed to package new versions of his bcachefs tools despite Debian team rejecting it, being a stable distro of course (source). Nick from "The Linux Experiment" has a summary of the situation that is worth watching (link). Brodie Robertson has a longer in-depth video for those who really want the full version (link).
I have the deepest respect for Kent's life work, but in my humble opinion, this situation cannot go on like this, as it is hurting the Linux ecosystem as a whole. Ext4 is old and has no protection against bitrot. ZFS is licence-incompatible to the Linux kernel and will therefore never be in-kernel, making updates a hassle and potentially hurting system stability because it really has to match the running Kernel version. Bcachefs is developed by mostly Kent himself, with sometimes help from others, for a long time in the shadows, now ready to emerge - and just as it did, has taken a severe double-hit by both D e b i a n (and therefore most likely all of its derivative distros like Ubuntu etc.) essentially stripping userspace-tool support for it, and L i n u s publicly releasing a statement that - quote - "Nobody sane uses bcachefs [and expects it to be stable, so every single user is an experimental site.] ", therefore making clear it does not have the necessary stability to be used yet (as opposed to the semi-stable, early adoption state that its developer often claims). Such a strong, clear wording by an institution like Linus Torvalds will definitely deter new testers that are willing to invest into early-adopting a new filesystem. The community has already spread that information far and wide on every newschannel as described above, most of you will likely have heard or read about it already by now.
In my opinion, it is now very likely that bcachefs' adoption by the distributions will stall. After such drama around the developer and a major distribution dropping support for it, I ask you now: Which major distro will include it even as an "experimental" option in an installer any time soon? And while I believe that people can learn, Kent's behaviour track record also contains the possibility that he might get dropped from the Kernel altogether should he continue to behave like this. Linus himself has hinted at that by choice of his words: "If bcachefs can't work sanely within the normal upstream kernel release schedule, maybe it shouldn't be in the normal upstream kernel. This is getting beyond ridiculous." Linus may be known for his occasional rants, but he also controls the Kernel development. So things don't look good for bcachefs at all.
We, on the other hand, have over 10 years of stabilization efforts under our belt and are widely adopted by several distributions and datacenters. The early growing pains are over, btrfs is stable now. The amount of developers working on btrfs is much higher, too, so we can get more stuff done in the same amount of time compared to bcachefs, which gives us the capability to make it feature-complete in the near future and then go for the big push: Replacement of ext4.
2. What do I propose?
The only two things we're still missing to get feature-parity with bcachefs (as described by Kents vision, not the file systems real current state which is much worse - it doesn't even have scrubbing yet) are:
By implementing both missing features, we can reach feature parity even to the mere "vision" or "idea" of bcachefs. Once that has happened, there will be no need for bcachefs and we avoid a flamewar between two filesystems, creating a situation where our proven track record will allow distributions to adopt us as the next-gen filesystem aka successor to ext4. (We could be called ext5 then, though I wouldn't change the name of course!)
3. What would happen to bcachefs?
That is up to the developer. Maybe he will continue development and we have a fallback / alternative / successor ready in 10 years. It's his life dream, he may work on it.
4. Why the need to "rush" or "push" for this?
We as a Linux community cannot afford to wait any longer. The switch to a next-gen filesystem (!) was supposed to happen ten years ago (!). We are risking falling behind other operating systems. The whole internet runs on Linux. We cannot afford to be replaced by other, better file systems from the properietary world (e.g. ReFS on Windows Servers, which supports tiered storage), just because we are content with the way btrfs is (Do we need tiered storage? Who uses raid 5/6?), or because we're secretly expecting bcachefs to succeed anyway and replace both ext4 and btrfs (as e.g. Kent hopes), waiting for a new project that will take ages to finish because of only one main developer, experimental stage, lack of testers, lack of distro adaptation - especially after the recent events around the developer. This was a long sentence, sorry, but you get the idea.
With SSD prices being and staying high in the forseeable future, on top of a rough economy in general, no one (not corporate, not small-business, not private) will want to refrain from using tiered storage if possible. They might choose another operating system entirely because of it. This has become mission-critical. Just my personal opinion, you may have a different one.
My point is that the Linux and IT landscape has changed a lot since btrfs was started in - looks up on Wikipedia, shocked face - 2007. Tiered storage is a must-have feature now. RAID5/6 might be conceptually difficult to implement safely because of the write hole, but we can do it. So many NAS and datacenters use it (because of the obvious space savings compared to other RAID modi) that we simply cannot avoid stabilizing it. These two features must be pushed as development priorities (next to bug fixes). Else, people will turn to bcachefs because it claims to offer these features and we basically will repeat the whole btrfs development cycle and it's gonna be the 2010's all over again!
In general, the switch from ext4 to btrfs that many people hoped for has never happened on a large enough scale. We are stuck, in a way, and need to work our way forward and make this project what it was intended to be.
If we don't try to push for feature-parity with Kent's project, people will want to wait for bcachefs to become mature (because we lack said features) and we will all (!) be stuck in a three-filesystem-situation (ext4 proven and stable but old, btrfs newer but feature-incomplete compared to the "next big thing", bcachefs feature-complete on paper but not ready for another 5 to 10 years) which means distro adaptation of the next Linux standard in 2030-2035.
We can't afford this situation to continue. Kent may continue his work in the background, but right now, he's slowing everything else down and allows other competitors in the industry to take our place. If he had managed to convince the Kernel team & the distro teams to push his fs as the new "ext5" candidate, rallying people behind him, the movement quickly gaining 100+ active developers working together as a team (instead of a one-man-show), having the necessary speed and documentation, and then actually doing the thing he wants to do, I would have pushed for bcachefs to become the next standard. This is nothing personal, of course. But with the current state of events, especially after the last news, he can be lucky if he doesn't get kicked out of the Kernel development and becomes unimportant.
We're the only project left that can do it. We're almost there. Linux needs btrfs to step up and finish what it has begun in 2007!
5. Any other recent reasons?
There seems to be, and it would be nice if you could relay this information to your friends & peers, apparently an almost exponentially growth in the number of people switching to Linux! (as Nick reported here)
Windows has made a critical error with introducing "Recall", people are starting to run away from it. These newcomers count on us. We can't afford to not have something modern ready! Be it desktops, hardware drivers, userspace software, documentation, as well as filesystems. This might be a once-in-a-lifetime opportunity to finally grow to a relevant number of people in society.
We should NOT put the Linux community through either: a) waiting forever in a confusing 3-filesystem-situation for a new standard to emerge, or b) repeating the early btrfs days all over again, just with bcachefs, losing data and trust in Linux as a whole
Btrfs early days was hard enough, people don't want another disappointment (5-10 years of unstable bcachefs) and they also don't want to fall behind on features. Linux needs btrfs to step up our game and finish what it has begun in 2007!
There is no other filesystem that can do it. We are the only ones who are still in the game, realistically, of becoming the next-gen Linux fs.
6. Any more reasons?
With more and more people believing the stories about "btrfs being unstable" and "bcachefs being the best filesystem ever, trust me bro" (and I have read a lot of these on social media), we will endanger a decade of stabilization efforts. I have read a lot of comments on e.g. Reddit from people that "still stick to ext4" because they simply do not see a valid successor in the Linux filesystem world emerge yet.
There seems to be some fanboi-ish situation going on. While this might not be intended by the developer(s) of bcachefs, it hurts Linux as an ecosystem because the trust in btrfs is being undermined by these social media comments. This might not be intentional, but its easy for people to tell each other stories from the early btrfs days and reflect them like in an echo chamber. Newcomers will then find and read these, and conclude to avoid btrfs. This is a situation we must avoid at all cost.
Implementing features and showing active development that is "on par" with what is expected to be the next big filesystem will help users see that we still intend to become the next standard and have not - and this might very well appear so to the outside world - abandoned our project.
TL;DR
This is a no brainer!
My appeal to all btrfs and btrfs-progs developers
I would therefore like to initiate a "focused-effort development phase" (read: form two teams, tackle the problems) of both
After a period of bugtesting (let's say whole of 2025), we should then finally push for ext4 replacement via the distros. They will accept us because of our proven track record over the past ten years, stable raid5/6, and feature parity to the other proposed next-gen file system on the horizon. Our code is stable and the two new features can be done, finishing the work.
A feature-complete btrfs will go a long way in restoring the community's trust in our project, and will send a strong signal that we are ready to take on the role btrfs was intended to fill when it was created.
Thanks for reading!
I am not a programmer myself, I wish I could help with that, but I can at least light the candle. 😎 In the hope that each and every one of you who can actually do these things now feels motivated to incite a group effort to do it. If we all work together we can do it! That's what Linux and Open Source are all about. We can do this.
I believe in the btrfs team so much that I have sent Linus Torvalds a link to this. Please don't leave me embarassed with an empty page, thank you! 😁