GwynethLlewelyn
commented
11 months ago A very long explanation on the causes of this issue
If you have nothing more important to read, and are genuinely interested in understanding more about the inner workings of the way Keybase is integrated with macOS, read on... I promise it's going to be very boring and long-winded!.
What is broken, and why?
Simply put, the integration of Keybase and Finder relies upon a technology known as Filesystem in Userspace (FUSE), originally developed for the Linux kernel. This is a technology that allows novel filesystems to be implemented safely outside the kernel, and presented to the underlying system as if they were 'normal' files — without requiring device drivers, kernel patches, extra services running, etc. — just like a 'normal' filesystem, so to speak. In fact, when well-implemented, the operating system is not even really aware of what exactly it's accessing or how the magic happens; a file is a file, after all.
It has been successfully ported to other Unixes, as well as to macOS (there are at least three variants; more on that below), and there is an equivalent technology for Windows known as the Dokan Library, which, these days, also features a FUSE-compatible layer. This means that (in theory!) if someone develops a new filesystem (such as Keybase, with KBFS) it can easily work, without change, across practically all platforms and architectures.
In practice, it's not that easy. Although FUSE support has been part of the Linux kernel for many, many years now, Apple frowns at anything not invented by them. And while Darwin (the operating system) is open source, the macOS kernel is not. However, Apple has generously allowed developers to 'plug' into the kernel the so-called 'kernel extensions', which technically run on a safe environment 'very close' to the kernel — but not too close! The variants of FUSE for macOS generally use a kernel extension to provide the necessary hooks for the FUSE API to work, and from that layer upwards, it's a standard FUSE API from the perspective of the developers. Apple doesn't like kernel extensions and has threatened to get rid of them in future versions of macOS (there are replacement mechanisms, known as 'system extensions', but they don't exactly replicate all the functionality), but, until Benjamin Fleischer, the developer/maintainer of FUSE for macOS, switches to the 'new' technology, we're stuck with some variant of his kernel extensions to get a working FUSE API.
Needless to say, from the point of view of Keybase (the company), this is exactly what they need to allow their own cryptographically-signed, network-based filesystem to be mounted on any operating system (including mobile phones!). They just need to worry about interconnecting between KBFS and FUSE. And then, each platform will have its own way of supplying the required FUSE API — and that is not something Keybase needs to worry much about any longer.
Now, in the Mac world, there is a catch. Every time there is a kernel upgrade, you need a kernel extension specially compiled for that specific kernel, and which might work on future versions — but will never work on older ones. This is due to the way Apple modifies its kernel and changes the APIs for the kernel extensions; they change (sometimes dramatically, when the architecture changes — from PowerPC to Intel and now ARM64) over the years, and that means that developers of kernel extensions have to keep them up to date with the 'latest and greatest' versions. Fortunately, in general, there is no need to specifically develop everything from scratch; in most scenarios, all that is required is just to recompile the extension with the Xcode SDK for the new version. Or hope that the old version somehow still works. You can use older SDKs to compile extensions for new kernels — strangely, that seems to be the most common scenario (also used by Keybase).
To make things more complex in the Mac universe: one of the first widespread kernel extensions, then known as 'Google MacFUSE for Mac OSX', was abandoned by its authors ca. 2009, but the source code was made freely available by Google. A new project was created by the aforementioned Benjamin Fleischer, it was renamed OSXFUSE, and was kept up to date (using the Linux kernel implementation as the reference).
Then Apple started getting paranoid with digital signatures for all installed software (except whatever you happened to compile yourself). Because those signatures require a paid certificate from Apple, it meant that many open-source project managers couldn't afford the luxury of signing all the releases; instead, they just supplied the open-source code, and we had to compile it on our own. This is tricky — it requires having Xcode (free) installed, with the appropriate SDK (also free), but potentially also a digital signature (not free!) and spending a lot of time tinkering with everything so that it works, spewing an installable binary after much munching and crunching over many minutes. Or hours, on old computers.
To side-step this issue, the team behind the OSXFUSE project made a dramatic change on their software base. While most of the code is still open source, there is something deep in the core — signed by the team — which is not, and it’s being distributed on binary form only. These components are still free to use, free to distribute, free to do whatever you wish to it (even resell it, if that's your plan), but limited to non-commercial usage: you just don't get the source code for it. Why? Because it was signed by the main developer himself, who cannot afford to 'give away' their keys to everybody. So they reached a compromise: continuing further development on the now rebranded "macFUSE" project, which is partially free and open-source, but has a closed-source component, only published in binary form, which is free for non-commercial use; while still keeping around (with no further development) the 'old' 100% free & open source OSXFUSE.
Apple, of course, went a notch further in their paranoia. Now it was not only necessary to digitally sign one's software; to be able to make it available on the App Store, or at least prevent macOS from complaining about 'unknown developers', the software, after being signed, has to be notarised by Apple — which appends their own digital signature only on software they 'approve'. I'm not sure if this has an extra cost (I believe not) but it obviously means that Apple will be quite aware of what is being distributed and how.
And on top of that, kernel extensions require a special digital signature from Apple (because they’re dangerous, after all!), which, allegedly, is quite hard to obtain. Benjamin Fleischer was lucky enough to be one of them, so his work under closed-sourced macFUSE could go on, while still being able to distribute the new kernel extension for free (for non-commercial use).
You can read a bit about the paradigm change on this article.
Keybase (the company) now faced a dilemma. They obviously could just get macFUSE being shipped with Keybase (both the open-source and the closed-source components) — possibly paying something to Fleischer. Since OSXFUSE doesn't get any further patches (and allegedly has some serious security issues — well, according to Homebrew, at least), macFUSE is now the 'preferred' version to enable FUSE under macOS. Also — and this might not have been obvious! — you just need one version of FUSE in your system: everything that uses FUSE to provide 'virtual' filesystems will 'talk' to whatever is providing the FUSE API. That would mean, for Keybase, that everybody who already had macFUSE installed on their systems (which is not that unlikely; it is quite useful and popular in many scenarios) would never need to worry about it. Others might just get a warning/notice to install it — or, better, if Keybase detected the presence of a Homebrew installation (possibly MacPorts also works; I haven't checked), it could automatically launch it to install macFUSE, and not worry about OS versions and architectures — that's the job of the Homebrew team!
Alas, the truth is that the number of users who are actively updating their preferred FUSE version is much smaller than the number of active Keybase users. And to make matters worse... many may have old/outdated versions of FUSE, or not the quite correct one, or just downloaded it from a different repository with its own installation quirks... and that meant Keybase wouldn't work. And they (the company) would have to give tech support over things they didn't control.
Instead, they opted to include their own FUSE kernel extension (a similar approach taken by other software developers, such as Atakama). But here they stumbled upon another problem: macFUSE, as said, has a closed-source component. With a digital signature. Now, to be able to distribute it with Keybase, the company would have to double-sign the whole package — or else, macOS (and the user!) would be confused, getting one package signed by two different developers. I'm aware that there is a way for creating a chain of signatures — ultimately notarised by Apple, of course — to avoid this double-developer issue, but it seems reasonable to assume that Keybase (the company) could not 'force' the developers of macFUSE to adapt their own software if and when required, while sticking to Keybase's own release schedule. They aren't Keybase employees, after all. (Never mind that, these days, macFUSE gets more regular releases than Keybase, but that wasn't true before 2020.)
Keybase's solution, therefore, required a single code source, that they could sign in toto, without requiring anyone outside the company to cooperate or collaborate in that process. This meant abandoning macFUSE, and embrace the venerable old OSXFUSE instead. Keybase doesn't do much with the code: essentially, they only do a search for the OSXFUSE tag inside the code and change it to KBFUSE instead. I believe that one or two patches are applied, to fix the most dreadful security issues in OSXFUSE, but, in general, the code is the same, and works exactly with the same API, and, most importantly, you can replace one by the other, and Keybase will still work. This is what they internally do, when they suspect that, somehow, their slightly-tweaked version of OSXFUSE doesn't seem to work in some scenarios, and try to use the original kernel extension instead, just to see where the problem is.
So, after all, where is the problem, then?
Shortly before Christmas 2022, Keybase (former?) developer @mmaxim was taking a look at some filed issues from Ventura users, who couldn't get Finder to mount the virtual KBFS with FUSE. He correctly figured out that OSXFUSE was not prepared to go beyond macOS 11 (Big Sur). This meant a change of plans, namely, adapting the macFUSE 4.X code to work with Keybase instead. How exactly that was supposed to work is beyond my understanding — at least, not without getting rid of the closed-source component — and I most certainly didn't check what exactly he did. The truth is that he managed to compile a kernel extension that worked under 12 and 13, thus making Ventura users very happy. He did make a mistake of some sort — a missing bundle — which was corrected minutes later. Then a release was officially produced and made available to anyone who had the app on auto-update and... @mmaxim disappeared from the face of the Earth, without trace.
Immediately, anyone not on macOS Ventura started to complain and post issue after issue, saying that the latest and greatest Keybase release was 'broken'. And it surely was. Be it by choice, or simply as a last-minute-patch-before-my-Christmas-vacation-from-which-I-might-not-come-back-so-soon, the truth is that @mmaxim only left a kernel extension available for macOS 12 (it just happens that 13 also works with the same kernel extension — but nothing that came before that!). All the other kernel extensions — as said, OSXFUSE supports everything from 10.5 upwards — were simply not included in the bundle, and I'm sure that this was pure neglect; after all, supporting all macOS versions will just require 160 Kbytes on disk (at least, the thin binaries).
Incidentally, @mmaxim also wanted to 'fix' the issues of those running Big Sur on Apple Silicon — alas, that didn't work as expected, since he 'forgot' to include a fat binary for macOS 11. Fortunately, it's presumable that Big Sur users on Apple Silicon have moved on to newer macOS versions (Monterey and Ventura, both supported) — until they upgraded to the Sonoma Beta, which won't work with Keybase's KBFUSE any longer (granted, Keybase has no obligation to support beta software, but Sonoma will be launched in a few months, so... it would be better to start updating the code now).
The disappearance of @mmaxim is actually a bit baffling. You might have noticed that @chrisnojima has been active as always, since early 2023 doubling his efforts to squash bugs left and right; however, he focuses mostly on the Electron GUI side of things. Maybe the main reason for the lack of an official release is the huge number of issues that nobody had time to reply to yet — we can only speculate on that.
So, now you know what was broken. How do we fix it?
The 'official' fix is not available to us
To do a proper fix, of course, requires recompiling the code, and this time, making sure that the KBFUSE bundle includes the missing kexts. In theory, this should just be a simple change. In practice, @mmaxim claims to have bumped the FUSE kext to '4.4.X' which is surprising — as said, this is supposed to be possible only with the closed-source component. I haven't checked what exactly has changed; the truth is that, on the source code tree, at the date of writing this,I can only find the binaries for the kernel extension. I can’t see where the source is. I have no idea if @mmaxim just copied the binaries from macFUSE and added them to the package. It looks like that for sure, but perhaps the actual source code is in a non-obvious place? If it isn’t, why should they bother to include so many string replacement scripts to change from “osxfuse”/"macFUSE" (with many different possible capitalisations) to “kbfuse”? Or is that just legacy code for the “old” OSXFUSE, which is not being called any longer?
kbfs is not supposed to support an external macFUSE 4.X (I tried that, to no avail). It either 'talks' to KBFUSE or to OSXFUSE (i.e. 3.X) only. Granted, it's possible that KBFUSE has, indeed, been bumped to 4.X, as well as kbfs, but that the code allowing external FUSE kexts to be used instead wasn't updated. That could also be worth investigating: I’ve peeked into the files and the code, and there is a ‘magic’ fsbundle.tgz — origin unknown — which is then somehow turned into a kbfuse.bundle (see below), using lots of search and replaces, but I haven’t (yet) come across the point where this fsbundle.tgz is procured — was it compiled on a previous step, or was it just copied over from macFUSE and patched? How did they manage to do that without breaking the cryptographic seal? I’ve noticed that the binary has most strings changed to “kbfuse”, but you can still find a few named “macFUSE” inside.
Or do they have already an agreement with Fleischer? After all, the building script assumes that there is source code “somewhere”: https://github.com/keybase/client/blob/4ce9e3e6826a372b3183dc94870a0e6af3b3663d/osx/Fuse/build.sh#L12
This source code package is not available to the general public, so one has to assume that it’s something downloaded separately from a ‘private’ website, to which Keybase employees have access, but not the general public.
In any case, this would require a full recompilation and a repackaging — or else, macOS will complain that the cryptographic seal had been broken.
Still, it's frustrating to see that perhaps less than 1% of the whole package needs to be changed (macFUSE still supports older versions of macOS, so, presumably, the source should be able to compile the kexts). If @mmaxim hadn't disappeared, I'm sure that he could fix his mistakes in a matter of minutes (and then, of course, take hours to compile & fully test all releases, even though most of that happens semi-automatically these days, not requiring him to stare in front of a computer screen). For an outsider, it means delving deeper into the codebase, and trying to understand how much can be salvaged to apply a fix. I also personally lack the ability to do cross-compilation to different versions of macOS or different architectures — which ultimately means that I could only do a version that happened to work on my Mac but couldn't guarantee that it would work on anybody else's Mac.
To make matters even more confusing... have you taken a look at what exactly the Keybase app contains? It's a little Russian Doll, layers within layers. Besides the Keybase.app proper — essentially an Electron executable — there are a few command-line tools which also get installed and put on the path. But there is more. Inside the app's contents there are two further apps, namely, an installer and an upgrader. These are full-blown macOS apps that get co-installed with the 'main' app. And it goes deeper: inside the installer app, there is the above-mentioned installable bundle, which installs kbfuse and the kernel extensions as well.
The dreaded error box saying that the Finder integration has failed is essentially communication between the 'main' app and its subsumed 'child' apps. Keybase, the main app, checks if it has kbfuse installed and connected to Finder; if not, it calls the installer, which, in turn, does similar checks, and unpacks the kext on its own, installs it in the proper place, and then writes three launchd configuration files on ~/Library/LaunchAgents — and calls launchctl to register & run them!
After that, when Keybase terminates, it cleans the configuration after itself, and asks to do the reverse (i.e. unregister the helper applications and remove the files for launchd). Those will always be regenerated every time Keybase.app is launched (manually or automatically, e.g. during login).
But this is not all that happens under the hood. A big question I had was — how does a userspace application acquire permissions to mount a (virtual) filesystem, without asking the user to authorise a privileged operation?
The answer wasn’t that obvious, but here Keybase is not especially original. What it also launches is a Helper daemon. This is a typical solution for a common problem. So, when a user gives permission to install Keybase, not only the kernel extension is plugged into its place, but the Helper software also gets permission to run in privileged (superuser) mode. That way, it will never be necessary to ask the user for permission: the Redirector, for instance, can run as the user, asking the Helper to make any necessary changes to mount the filesystem on its customary place (e.g. /Volumes/Keybase ([[[YOUR USERNAME HERE]]])). That’s another clever trick!
Anyway, if any of the above fails, then Keybase app gets 'confused' and doesn't quite know if Finder mounted the volume or not. When it attempts to do something on the KBFS filesystem, and it doesn’t work, then it tries again. Sometimes, it might even trigger a new attempt to call launchctl and try to get the other daemons active. Sometimes, however, it’s Finder that believes it has a mounted filesystem — because FUSE tells it that it’s there — but obviously ‘nothing works’, so Finder will fail — often entering an endless loop and not even recovering when it gets force-quitted.
How this ought to work (but doesn’t)
Keybase, in its many variants (OS & architecture), is usually configured from several sources:
- Built-in variables (which require a recompilation to change)
- Command-line arguments (which are specific to each daemon; they won’t work across them)
- Configuration files on disk (usually things called
config.jsonfor the overall setup andgui_config.jsonfor the GUI-specific aspects, as well asupdater.json, all of which, under macOS, are normally stored under~/Library/Application Support/Keybase, where they can theoretically be edited by the user) - Environment variables, which allow quick overriding of some parameters, and side-step the need to change any files; Keybase actually uses a few to pass parameters to the
kbfsdaemon
While most of the command-line arguments can be seen with keybase help, this mostly applies just to the ‘main’ Keybase core app. The other tools, usually residing where the user cannot get at them (lest they break the cryptographic seal), are much more sparse in information — you can get some of their arguments, though, and this is especially true of /Applications/Keybase.app/Contents/SharedSupport/bin/kbfs -helpwhich will give you quite a lot of parameters to tinker with.
The most important, though, for our scenario, is -use-system-fuse. As its name implies, this allows one to launch kbfs using a previously installed FUSE kernel extension, as opposed to using Keybase’s own. In the source code, there are scattered comments explaining that this was intended for ‘testing purposes’ — namely, giving developers the ability to test dfferent versions of macFUSE (or OSXFUSE, or whatever comes next) without requiring a full recompilation, targetting the ‘new’ FUSE library under experiment.
Therefore, the overall solution for this issue should be quite simple: just install your preferred FUSE extension (kernel or even possibly a modern so-called ‘system extension’), figure out where it’s installed (easy if you did it via Homebrew!), and pass that parameter to kbfs when it gets launched.
That’s what sensible programmers would have done, right?
And probably they did, but without any sort of documentation (not even any clue where on the source code this may be called), it’s next-to-impossible to figure out where such parameters can be changed by the user. I tried the obvious — changing config.json or adding some environment variables, but the problem is that I don’t know how they are called/named (Keybase refactored part of the code to use a ‘standard’ naming convention, but this neither applies to the ‘older’ flags, nor is it exhaustive — many things still rely on built-in variables which cannot be changed without a full recompilation & digital signing).
Worse than that, I couldn’t even understand where exactly those parameters might be stored on disk. While it’s easy to figure out how things work for the keybase core application (i.e. change config.json — at your risk, mind you), kbfs doesn’t appear to have an equivalent external configuration file, nor a check for environment variables — although I cannot truly be sure of that. Maybe the code is there, maybe not. Maybe, even if it’s not there, it can be easily copied from the other tools. Too many maybes!
However, it’s naturally easy to see what parameters kbfs requires — that’s trivially found by simply doing a ps xaw | grep kbfs and the parameters are simple enough. It took me a while to figure out where Keybase actually launches kbfs with those parameters, because I couldn’t find them anywhere in the source code.
As you know by now, these parameters are not there at all, because kbfs is not called directly by keybase. Rather, as you have seen, keybase writes those files for launchctl to launch kbfs as a ‘standard’ daemon under control of launchd. One might argue if this is the ‘best’ way of doing things or not. I’d argue it is the most effective way of avoiding to break the meticulous logic of the way macOS works, but I fail to understand why the *.plist files are created on demand and deleted afterwards. That makes no sense to me. The more reasonable approach would be, during installation, generate those files and place them where they should be (~/Library/LaunchAgents, of course) and keep them there.
Maybe Keybase (the company) has a good reason for doing things this way, but I fail to understand it.
What I can say is that ~/Library/LaunchAgents/keybase.kbfs.plist most certainly contains the ‘right’ parameters for launching kbfs. I was a bit baffled by the ‘requirement’ for always running in debug mode, spewing miles-long logs all the time (which is never good in production!). It looked obvious that this was a flag they added during development, but forgot to remove it once deployed.
Again, one would assume that these parameters are read from somewhere (disk, environment, possibly even command-line arguments to keybase which would then be passed to the function that generates the .plist file), but no. I have checked the source code: they are hard-coded. They might be overridden somewhere — I couldn’t find it — but the bit that actually writes out the .plist has no variables — just explicit strings. Without any further clues (better documentation for developers!), I certainly cannot find an alternative that doesn’t require compilation.
So, at this point, I was stuck: I couldn’t find any place to configure kbfs persistently so that it communicates with a different FUSE kernel extension. I could, obviously, test if it works, by shutting Keybase down, injecting ‘my’ FUSE kext into the kernel, and launch kbfs with -use-system-fuse. That works! But without being authenticated to Keybase, this does nothing (except writing tons of logs).
One tiny little ray of hope was the following experiment: what if I launch kbfs with ‘my’ parameters first, and then launch Keybase, to get authenticated?
And, in fact, during a tiny slice of time (at around 6 AM my time, after looking for a solution for several hours), kbfs did receive the required authentication, and it did ‘talk’ to the Keybase servers — and, aye, the virtual filesystem was mounted, and my files started appearing in it... even under Finder!
Yay?
No, not really. A fraction of a second later, Keybase writes ~/Library/LaunchAgents/keybase.kbfs.plist to disk and invokes launchctl to launch kbfs with its own set of parameters. The two daemons struggle for a while for ownership of the filesystem. However, Keybase’s launched kbfs can cheat — it runs from launchd and has KeepAlive set. What that means is that launchd will attempt to launch and relaunch kbfs, incessantly, until it gets the confirmation that the daemon is operational. And under some circumstances, it kills all instances of kbfs (even those that aren’t under its direct control), to guarantee that, at last, its ‘own’ daemon can run, without interference.
While this is terribly annoying in our scenario, consider how Apple needs to give technical support to clueless users. Apple doesn’t really want to deal with complex configuration problems: macOS, after all, should ‘work out of the box’ and be tamper-proof, even by the legitimate owner of the Mac. This behaviour, therefore, makes sense on an operating system that is supposed not to allows users (legitimate or malicious) to tinker too much with it.
It’s also of no avail to attempt to kill the kbfs daemon that is launched via Keybase, while only keeping my own version running. You can do that, but launchd while run it again, if it detects that it’s not running (and that it should). This happens very quickly — I can’t even use a stopwatch to tell you how fast it is, but you can safely assume that, from the perspective of the user, it is as if it were instantaneous .
The kludge
My ‘hack’, therefore, was quite simple — since I had no other options left.
If Keybase only let the keybase.kbfs.plist in place instead of constantly deleting it when Keybase is shut down, while generated freshly every time Keybase starts, then it would be an easy fix: just add the extra line for -use-system-fuse, and let launchctl do its work.
However, the ‘cleaning up’ after leaving Keybase makes some sense. One presumes that if the application is shut down (‘completely shut down’), users expect that nothing else will be running (well, eventually, an exception can be made for the tiny FUSE kext, which has to be explicitly removed, but if it’s not being actively called, it won’t consume any resources). That means not only no Keybase, but also no Helper applications, no KBFS, no mounted volumes, nothing.
There are obviously ways for doing that without having to delete every configuration file for launchd; I’m aware of at least two mechanisms to do so, and the simplest one is just to call launchctl with the disable argument. That’s all it takes.
Why exactly Keybase didn’t go that route is beyond my understanding. I thought it could be something related to users not wanting to run Keybase at login — but that’s not even an issue with launchd, which can be configured to be activated on demand or every time the user logs in; Keybase (the app) doesn’t need to worry about any of that, just make sure that the configuration file is added to the correct ‘domain-target’ (in Applespeak). And if the user checks the box to launch Keybase at start, then the configuration file just needs to be moved (logically, not physically!) to a different domain-target. That seems easy enough for me.
It would also obviously allow users to change that configuration file 😉 (which is, after all, the whole point of placing these files under a user’s Library!). Maybe Keybase really, really, really doesn’t want anybody tampering with that file?
Well... I don’t want Keybase to tamper with my files, either!
After a few failed experiments — the file gets overwritten even if it has no write permissions, or even if it’s owned by root — there seemed just a way to keep the file there: make it immutable.
For those that aren’t aware of the many layers of protection on the Mac filesystem, there are at least three I’m aware of. The first level is common to all Unixes, i.e. the file permissions that can be set to owner/group/all. Since any user can become a privileged user, and that means giving them the ability to break anything (or everything!) beyond repair, e.g. sudo rm -rf / (the kind of command that you do not want to experiment just to see if it works!!).
On top of that ‘basic’ level, Apple has implemented file attributes, available using the xattr command. This allows any file to get lots of metadata, some of which with the ability to be semantically interpreted by macOS in subtle ways, such as the ‘infamous’ quarantine tag. Others are very specific to certain applications and might not even be documented; after all, any macOS application can create whatever metadata they want, and, so long it doesn’t conflict with existing tags, give it whatever ‘meaning’ they wish.
But there is another layer of protection, which Apple inherited from BSD 4.4, and which are known as flags. These apparently are around since 2006 (possibly earlier!), and can be viewed with /bin/ls -lO and set/changed with chflags (which is a command that has a man page). Their usage is a bit esoteric, but two at least are quite useful: one allows that file only to be appended to, not rewritten (useful for having secure logs, which can only grow, but there is no way of deleting earlier entries); and another is the immutable flag, which, as it name implies, means that this file can be read, but nothing else is allowed — it cannot be changed in any way, nor even moved from its place. Some key operating system files are set to immutable to make sure that not even the superuser can change them (and thus prevent anyone from causally tampering with them); while it’s not uncommon for regular users to set some of their files immutable so that they cannot be ‘accidentally’ removed with a careless rm -rf.
Although the 2006 man page cautions the user that not all utilities respect these BSD flags (and can therefore override them), the good news is that, whatever Keybase uses to generate the launchd configuration files, it does respect the BSD flags (it could be one of those things built into the Go library handling I/O — which present a common interface to the programmer, but are implemented differently on each platform that Go is compiled on).
WIth that ‘trick’ — setting keybase.kbfs.plist to immutable — Keybase cannot ‘hurt’ our configuration any longer. But it will try! That’s why I point out the possible conflicts that may happen, especially on the first attempt at launching Keybase, with kbfs already running with our configuration instead. Keybase will try to remove the file and recreate it. It will launch kbfs, sidestepping launchctl, and forcing its own configuration. We human users just need to be more stubborn and persistent than Keybase, killing whatever extraneous processes it launches against our will. Eventually, launchd will step in and only allow our hacked file to be used — possibly killing some kbfs that has been launched outside its control — and the overall setup will be stable for a while. Keybase will have no other choice left but to talk to our instance of kbfs, which, in turn, will only communicate with OSXFUSE (and not with Keybase’s own KBFUSE), even if both kexts are plugged into the kernel (which they shouldn’t, but weirder things have happened...).
There are, of course, limitations. kbfs will run all the time — no matter if Keybase has been launched or not. Since kbfs lacks the required authentication provided by the Keybase app, it will endlessly complain about it on the logs, all the time, which is not a really good thing to have. At least, the -debug flag can be removed, which limits the amount of excessively detailed entries on the logs.
And, aye, this is not 100% stable — not by far! Every now and then, Keybase stops communicating with kbfs, or revokes its authorisation, or something even more weird happens — and the connection to the remote folders is in an inconsistent state. Sometimes it’s possible to resolve the issue by liberally killing kbfs from the command line, and possibly forcing the keybase.kbfs.plist to be re-read/re-enabled via launchctl, thus restoring its previous state. Sometimes Keybase must be shut down during this operation; sometimes it doesn’t care (after all, it’s not supposed to do anything with kbfs, except for checking, at launch time, if kbfs is running and set it up if not). And sometimes Finder crashes and cannot recover. Sometimes a reboot is the only way to fix things.
These are inconveniences, especially if you use the KBFS actively, i.e. with some kind of application directly writing on it (such as, say, Notes...) — because when the connection goes down, you might lose your work. An application reading/writing to a file on KBFS at the moment it loses connection will very likely enter an infinite loop, expecting the connection to be up again. Re-launching kbfs will re-establish the connection at a high level, but lower-level file descriptors will be lost — and that means the application will almost certainly have to be force-quit (thus losing all the work being done...)
Possible future solutions
I’m not going to delve deep into this. My hope is that someone at Keybase at least takes notice of this issue, and, as a consequence, decides to either revert commit #25338, or, in a much more reasonable way, allow us to modify many more configuration parameters.
One thing that has intrigued me is how the latest Keybase versions are able to ship a version of 4.X, which allegedly includes the issue with the digital signature affixed on a closed-source component — because OSXFUSE does not support (directly, at least) Apple Silicon, only macFUSE 4.X does that. But kbfs allegedly refuses to connect to anything except KBFUSE and OSXFUSE — at least, that’s what I gather from this: https://github.com/keybase/client/blob/4ce9e3e6826a372b3183dc94870a0e6af3b3663d/go/kbfs/libfuse/mounter_osx.go#L86 and this: https://github.com/keybase/client/blob/4ce9e3e6826a372b3183dc94870a0e6af3b3663d/go/kbfs/libfuse/mounter_osx.go#L39
There are not many references to how exactly kbfs ‘knows’ that it’s talking to an ‘old’ version of the FUSE kext — which it supports — while (allegedly) refusing to ‘talk’ to anything more recent. But... isn’t kbfuse supposed to be 4.4.X already? How does kbfs reject third-party FUSE kexts above 4+, while at the same time allowing kbfuse to run?
Even weirder... to ‘talk’ to the FUSE kext, Keybase uses the popular Go library from bazil.org/fuse. The maintainer of the bazil-fuse project, @tv42, seems to have dropped all macOS support on his library: https://github.com/bazil/fuse/issues/224
While I couldn’t test it myself, however, it seems that ‘pre-3.3.0’ support is still in bazil-fuse. Or else, how could Keybase use it? Unless, of course, if they are just using an old version of bazil-fuse (and don’t dare to upgrade it!).
Not surprisingly, this dilemma is faced by many other open-source projects that use macFUSE. Here is an interesting discussion/debate between the developers of plexdrive and macFUSE’s own Benjamin Fleischer: https://github.com/plexdrive/plexdrive/issues/390
The bazil-fuse maintainer, however, admitted back in 2016 that there might be a compromise, since recent (i.e. 4+) versions of macFUSE do, indeed, support a very complex mounting mechanism, which apparently works similarly in both macOS and Linux. I’m afraid this is way too deep-level in the respective kernels (which are so different!), but I can understand that, over the years, thanks to some convergence of efforts, there are now some more common (or at least more familiar) kernel APIs. They must be different at some level, though, because, as said, the kernels are completely different beasts. FreeBSD inherits the same codebase (BSD 4.4) as macOS/Darwin, but there are radical differences between both: Darwin uses a microkernel architecture based on Mach (XNU, to be more precise), something that Apple inherited from Steve Jobs’ work back when he founded NeXT. FreeBSD, by contrast, has a monolithic kernel — in fact, the same approach taken by Linux, which comes from a completely different family (namely, SysV Unix).
Interestingly, there was some convergence between the Linux and FreeBSD kernels, mostly because of the Internet: the TCP/IP support on BSD was far superior to anything else in the mid-1990s, and, since it was free and open source, even Microsoft Windows used it (seriously!). Linux started using its own solution, but BSD’s was by far more superior. This led to the kernel developers from Linux and FreeBSD to come together and adapt their kernels so that many things are similar, even if their origin is completely different (and it shows!). What this means is that, in some regards, the “BSD philosophy” is followed by the Linux kernel, and this has become familiar to those who use Linux only, and are scared of macOS/Darwin — which also uses the “BSD philosophy”, just a conceptually totally different kernel.
Obviously, there is the easy way out: dump bazil-fuse altogether, use go-fuse instead, and let the users decide what ‘flavour’ of kernel extension they prefer. Go-fuse supports them all, and their maintainers are trying to essentially support all FUSE implementations they can.
Needless to say, this is not a drop-in replacement; a lot of code would have to be refactored to take it into account.
Anyway, I digress (but this was the “futurology” section, so I’ve got an excuse!). I really didn’t delve that deep in Keybase’s December patch — just merely scratched the surface — and therefore haven’t reached a conclusion yet. One thing is for sure: without a bit more help from the Keybase developers, it’s hard to ‘fix’ the code.
My best suggestion, therefore, would be to specifically target kbfs, replace bazil-fuse with go-fuse (a huge endeavour!), and allow it to ‘talk’ to a ‘genuine’ macFUSE kernel extension, installed separately from Keybase. Then, with my kludge/hack/whatever, you can prevent Keybase to start its own kbfs, closely tied to whatever their developers think the ‘best’ kext is, and just go ahead with ‘universal’ support from both go-fuse and, of course, macFUSE.
Because everything else can remain in place: the Keybase application doesn’t care what kbfs is doing, deep below. It just checks for its presence when it starts. Admittedly, it would be wonderful if Keybase allowed that step to be overriden with an environment variable or a command-line option, or even an entry in the config.json file. That would take care of all issues: provide a bundled version including all the Keybase-signed binaries, the GUI, and their specially-modified kernel extension, which might have limited support for some macOS/architecture combinations; while still allowing this installation step to be skipped over, and just let people install whatever they like to get Keybase to play nicely with their Macs.
Or, alternatively, change the kbfs code to do authentication on its own (it’s harder than it seems), not relying on anything except the Keybase Helper application — to be able to mount the virtual filesystem on the proper mount point. In other words: essentially providing the same kind of experience under macOS as what Linux users get — an optional GUI-less environment that allows them to authenticate with their logins & passwords, and then proceed to properly mount he remote filesystem locally. Technically, you could even get rid of both the Helper and the Resolver — just give the user the decision of selecting their own mount point — personally, I find that ~/keybase/private/myuseername/... is easier to type than /Volumes/\ \(myusername\)/private/mysusername...’, but clearly Keybase thinks otherwise ...
Thanks for reading till the very end!
Digitally yours,
— Gwyn
Note: on the comments below, I was warned (correctly) that certain expressions in this text could be considered offensive and drama-inciting, and therefore I deleted or changed them (they didn't really add anything to the overall context). There was no intention to offend anyone, of course, much less those who have kept their projects around for years and years and having hundreds of thousands or millions of users relying upon their code, often published as free and open-source. I'm fond of hyperbole, but I agree that this time I should have been more careful, more neutral, and, especially, pay more attention to the real human beings who live behind their nicknames. Nobody benefitted from those expressions and turns of phrase, so I have eliminated them all, as far as I could tell; if you find anything that is still offensive, drama-inciting, exaggerated, wrongly explained or presented, or simply factually wrong, please let me know. Like so many of you, the only thing I want is to get a fix, a patch, something to keep everything working; unfortunately, I'm terribly limited in my knowledge in how things work at a deep level, close to the kernel, and don't even know where to start.
In any case, I'm still more inclined to believe that it's the Keybase code that should be changed — not the FUSE extensions for Go, or the kernel extensions injected into the macOS kernel — but that would definitely require a repackaging and a profound refactoring on how the application is distributed/published in the macOS ecosystem. That is only up to Keybase (the company) to deal with, not those who created all those pieces of software, without which Keybase's KBFS would never work — and it was quite offensive for me to direct my rants to them, who have absolutely nothing to do with Keybase and Keybase's decisions on how their software is installed.
Again, my apologies; I'll endeavour to be more careful and more respectful in the future.
macos-fuse-t
anacrolix
jacksongoode
tv42
vinyll
seanPhill
Hello all,
After seven frustrating months without the ability to use the Finder integration on macOS Big Sur (currently at 11.7.9 (20G1426)), I finally found a workaround! I'm posting this here because every month someone complains about the integration of Keybase with Finder not working any longer with either very old or very new versions of macOS.
TL;DR (well, not really, the workaround has many steps to follow)
(way longer explanations on the comment below, if you really wish to know more)
The issue
In order for Keybase to mount KBFS as a 'regular' filesystem (as viewed by Finder), it uses Filesystem in Userspace (FUSE). This requires a kernel extension to be installed, and every version of macOS requires at least a recompilation of that extension in order to work in subsequent versions. You can grab FUSE from elsewhere, but for several reasons, Keybase bundles its own version.
Unfortunately, on Dec 12, 2022, @mmaxim, in an attempt to get a quick fix for macOS Ventura users, published, in quick succession, the commits #25335 and #25338. These fixed the issue for Ventura, while breaking it for essentially everybody else. @mmaxim has been strangely absent — he doesn't reply to messages and has barely committed anything during 2023 (and nothing related to Keybase). As far as we can publicly see, the last time he logged in to GitHub was back in March. @chrisnojima, who publishes at least one new commit every day (sometimes, even more than one), is apparently unaware of the Big Mess that @mmaxim left behind, or simply hasn't got time to do a real fix.
Partial list of issues filed from users affected with this problem
Some are directly related, others indirectly; the list still continues to be expanded. If you've seen the following popup:
then you're welcome to the club! You can see it's not very exclusive — lots of people have experienced exactly the same issue (and still are!) and are waiting for a fix. Here is a sample:
- #24366 (the first report, on early November 2022)
- #24376
- #25043
- #25130
- #25226
- #25263
- #25271
- #25294
- #25297
- #25321
- #25328
- #25333
- #25379
- #25452
- #25486
- #25510
- #25526
- #25632
- #25637
- #25718
- #25729
- #25729
- #25767
- #25770
*Note: the above list may be very incomplete, I essentially followed a chain of issues that referred to each other; there might be many more who never made any reference — meaning that the author of the issue never bothered to search here — and thus I couldn't find it. But it should give the Keybase developers some pause for thought.*
The only known workaround so far is to install a previous version of Keybase, in the hope that it will still bring the "old" Keybase FUSE port which still works with older kernels (as it should). The newer versions of Keybase will always overwrite any attempts of "fixing" the Keybase FUSE kernel extension (by replacing it with a working one); thus my kludgy workaround below.
Going 'back in time' to an old version may not be always possible, of course, and that's why I'm suggesting a very kludgy workaround, which should work with every future version of Keybase (I'm testing it with the nightly builds).
All that the Keybase devs need to do is to partially revert the changes made in early December 2022, keeping the old kernel support in, while adding support for newer kernels (just like the current macFUSE version does).
My kludgy workaround
I have found a very, very kludgy workaround for those who simply cannot live without this feature (myself included!) and would give an arm and a leg just to get it fixed.
Fair warning: this is not for the faint of heart. You might break everything (I didn't, but I guess I was lucky) and never be able to use Keybase again. Or even your Mac. I don't know, and I won't help you out if you have a problem; accept this solution 'as is' and don't complain if it bricked your Mac — you're doing this on your sole responsibility, and, who knows, you might not even get Apple to fix your Mac during the warranty time. Caveat utilitor.
That said, I'm not sure if this will work on anything but macOS Big Sur, because that's the version I've got. However, I'm pretty sure that the following macOS versions should work: 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11, 10.12, 10.13, 10.14, 10.15, 10.16, 11.X. Anything above that... well, I cannot say, but at least the latest Keybase version works 'out of the box' with 12 and 13. You're out of luck with 14, though.
cd ~/Library/LaunchAgents/. There should be three files there, starting withkeybase...keybase.kbfs.plist: e.g.cp keybase.kbfs.plist ~/Desktop(or wherever you prefer).ps xaww | grep -i Keybasea few times, just to see if some process is lingering (and, if it is, force-quit it from the shell prompt with akill -9 <pid>, or even withsudo kill -9 <pid>).~/Desktop/keybase.kbfs.plist.There might be some slight variants, but it should essentially be what's above. 8. Locate
<key>ProgramArguments</key>, and, below, inside the<array>, you can see the parameters being passed tokbfs. We want to add a new parameter, right at the start. Just before<string>-debug</string>, add a new line with<string>-use-system-fuse</string>. Save it! Your file should now look like this:cat ~/Desktop/keybase.kbfs.plistand check that no weird characters have been added. It should be a pure ASCII file. If you're paranoid, you can even do afile ~/Desktop/keybase.kbfs.plistwhich should sayexported SGML document text, ASCII text. The part that is important is theASCII text. If you getdataorbinaryorRich Text Formatinstead, then you have probably inserted some characters you shouldn't; either fix it or delete it and restart from scratch (i.e., back to 1.).cp ~/Desktop/keybase.kbfs.plist ~/Library/LaunchAgents/. Confirm that it's in the right place!chflags uimmutable ~/Library/LaunchAgents/keybase.kbfs.plist. Check that this worked properly (the dollar sign is your prompt, do not copy & paste it by mistake):The crucial flag there is the
uchg(which means unchangeable). So long as that flag is there, nobody can touch that file. Ever. Not even the superuser! You can try that on your own —sudo rm -f ~/Library/LaunchAgents/keybase.kbfs.plistshould give an error. (If you wish to know more about what this does, you'll have to read it on the long version below.) Nevermind the date or the file size, they will obviously be different on your system.Ok, this got rid of Keybase's own FUSE package. Now we have to install the 'correct' kernel extension for FUSE:
brew install osxfuse. Ignore ifbrewcomplains about security issues and nags you to install the much more recent version, now rebrandedmacFUSE. Ignore it. You want OSXFUSE. I believe that the version bundled by Homebrew is 3.11.2.sw_vers -productVersionfrom the shell, if you prefer. If you're on Big Sur, for instance, you should have something like 11.7.9.osxfuse; note that Homebrew might give different results for different versions of macOS and/or architectures. Here is what I have in an Intel MacBook Pro (mid-2014) with Big Sur:You wish to use the version that matches your version of macOS (note that sometimes kernel extensions work across versions, thus the symbolic links). 5. Now load the kernel extension. Cross your fingers! For instance, if your macOS version is 10.12, then you should run the following command:
Apple Silicon (ARM-based) Macs may need to follow further instructions. For your reference (in case of link rot), this is what Apple's manual says:
Also note (if you have Apple Silicon) that the Keybase-provided FUSE kernel extension is a 'fat binary' supporting Intel and ARM64. Homebrew apparently ships the extensions only for whatever architecture you've got, but it's worth figuring out if that's true: if you do a
file /Library/Filesystems/osxfuse.fs/Contents/Extensions/10.12/osxfuse.kext/Contents/MacOS/osxfuse, it should tell you what architecture it has been compiled for. Under my old Intel Mac, Homebrew just provided me with a 'thin' binary for x86_64; YMMV. 6. Check if it was properly loaded:Again, what matters here is that you actually get that line displayed — numbers, etc, will be different. 7. If your macOS version doesn't have the
kmutilcommand, you can use the (deprecated)kextload(instead ofkmutil load) andkextstat -b(instead ofkmutil showloaded --bundle-identifier).All right! Now everything should be in place. Most people will require a reboot at this stage, others will have some luck merely by launching Keybase...
If you do a reboot, the first thing to check is if the kext has been properly loaded, using the command shown on 6. above. This ought to be working.
After a reboot, some magic should have happened, even before you start Keybase. You see, because we left the immutable file on ~/Library/LaunchAgents, the system launcher ought to have launched
kbfs, the Keybase FileSystem daemon, which provides the 'bridge' between Keybase, Finder, and the FUSE kernel extension. You can check that it's running:If you do look at the logs, you will see a lot of errors, which is to be expected: Keybase hasn't been launched yet, and that means no authentication was performed, thus
kbfscannot do anything until that happens.You should also now get a new icon on the Preferences Pane, at the bottom, saying "FUSE". When you click on it, it should tell you the version of FUSE that has been installed by Homebrew. Do not click on update. I know it's tempting, but you do not want a different version — not until Keybase releases a permanent fix.
It is also possible that at some stage you'll encounter a popup like this one:
It might not be exactly that (I forgot to take a snapshot, so I'm reusing an image from a different context), but the purpose will be the same: authorise your Mac to run a kernel extension signed by Benjamin Fleischer, a legitimate Apple-approved developer, who has been doing versions of FUSE for Mac for many, many years. The box is a bit cryptic, and sometimes it's not easy to switch to the Preferences and click on the right place that gives permission to run the kernel extension. Or sometimes you have it all greyed out and have to click on the lock to be allowed to make any changes... and when you do that, the message to allow the extension to be run disappears! It's terrible, I know, but the alternative is to give permission via the command line instead:
Why 3T5GSNBU6W? Well, that 'magic number' actually identifies Benjamin Fleischer — that's his 'Team ID' as registered by Apple. If you wish to see who else has registered Team IDs on your system, you can see them all here:
It shouldn't surprise you to see Keybase, Inc. listed there as well, with the Team ID 99229SGT5K.
Note that this is basically just a list of extensions that are allowed to run; they aren't necessarily active (with the command above, the third column should show extensions allowed to run — 1 — and those that have been at some point installed in the system but aren't allowed to run any more — 0). Therefore, it doesn't matter if you have multiple versions or variants of FUSE for Mac on that list (including the one from Keybase!). What matters is that the one we want to run has permissions to do so.
All right, everything is in place now — it's time to start Keybase!
A lot of things will happen, most of which will be very confusing to follow, especially if you bother to look at the logs. What Keybase will now try to do is to change the immutable file to launch its own kernel extension instead; this will give some errors (possibly on the logs). Since we have our version of
kbfscorrectly configured to 'talk' to the correct kernel extension, Keybase won't be able to start its own — it will try, though, and get impossibly confused, as now twokbfsare running on the system, one of which non-operational. In one circumstance, Keybase even attempted to get my permission to reinstall the kernel extension, hoping to 'fix' the mess (according to its perspective); I allowed it to do so, confident that it couldn't do much about the immutable file — and, indeed, that's exactly what happened.In my case, during this 'crash of titans', 'our' tweaked version of
kbfsmay shortly notice that it has been granted authentication to connect to Keybase's remote filesystem, and start happily to tell that to Finder. This might only last a few moments, though, because the 'flawed'kbfsinstance will try to 'steal' that authentication for itself, and both will compete to get hold of Finder (potentially crashing it, BTW). Anyway, while it's fun to watch (with the logs open, no less!), it's not what we wish to have, so the trick is to put an end to it all, and this is accomplished simply with(you can even do a
sudo killall -9 kbfsfor good measure)What happens next is actually also amusing. Of course, when
kbfsdies — all instances of it — your Finder access will not work with the remote folder, and, again, it's not impossible that Finder won't crash again. However, there is now a difference: the mechanism that launcheskbfsis not Keybase, but rather the system's launcher, which is set to launchkbfsover and over again, every time it 'dies' for whatever reason. Keybase, therefore, doesn't interfere with the process any more — it knows that, sooner or later (usually sooner!),kbfswill be up again. Also,kbfsis a helper function for Finder — as you have noticed, during all these past few months, you could always access your files, either from the GUI (under Files) or from the command-line (withkeybase fs (...)series of commands). It's only Finder that needskbfs, so Keybase leaves it in peace.Eventually, therefore, the system launcher will re-launch
kbfsagain — but this time, it only has our immutable file to load, and therefore it will cleanly launchkbfsconnected to the correct kernel extension — and Finder will be immediately aware of the service!In theory, this is something you only need to do once — possibly killing
kbfsagain after you install a newer version of Keybase, which might try to run its own installation script again and fight for the right to bind to Finder. In practice, I would guess that every now and then you will need to happily kill some processes, restart Keybase, or even reboot, in order to fix things. I've noticed some inconsistent behaviour after 'waking up' the PowerBook which had been in 'deep sleep'. Also, if you close Keybase (or even just the GUI) and start it again, it might trigger the conflicts once more.If by any chance Keybase kicks your 'tweaked'
kbfsout of the process list, and tries to launch its own untweaked version, then you can try doing the following:Eventually, one of the last three commands will kick the 'correct'
kbfsinto place. Be prepared to do more frequent reboots, though; this hacky configuration may leave Finder unresponsive (as warned earlier) and utterly confused about what mounts are still valid; while sometimes Force Quit on Finder will work, other times you have no choice but to reboot.As said in the very beginning — I told you that this was a kludgy workaround, right? 😸
There you go. That's all. The rest is reference material for those really interested in more information, and how I figured out this very crude and ugly way of 'fixing' things.
Oh, and make sure you send @mmaxim a few messages, asking him to revert the mess he did in December 2022.
(And I apologise for the length of this issue; there are simply way too many steps to take until it works)