GUI and KBFS bugs in 5.9.0 on Windows and Linux

[paddyredbeard]

Since upgrading to 5.9.0 on Windows 11, keybase does not fully start. The GUI is stuck on a Loading message and kbfs is not mapped to K: or any other drive letter. This would seem to be the same issue as reported in https://github.com/keybase/client/issues/24745 and https://github.com/keybase/keybase-issues/issues/4087.

I have also experienced a similar issue using the linux client (for debian-based systems). Upon upgrading to 5.9.0, kbfs would not cleanly mount at /keybase, though the GUI did seem to work as expected, and I found my keybase files could be accessed at /run/user/<uid>/keybase/kbfs.

On linux I was able to downgrade to 5.8.1 and set an apt hold to prevent re-upgrading to 5.9.0.

my log id: 23ad4baebe37f3d92088101c

[paddyredbeard]

Note: For debian-based installations, here is the workaround to downgrade, for those who may be interested...

1. Go to https://prerelease.keybase.io/ and then click the linux/deb link
2. Download the version 5.8.1 .deb file appropriate for your system (ie. the i386.deb file for 32-bit systems or the amd64.deb for 64-bit systems). Optionally (preferably) also download the corresponding PGP signature and verify the downloaded .deb file
3. Run sudo apt purge keybase to remove version 5.9.0. You may need to reboot at this point (on one system I needed to do so to cleanly install 5.8.1 but not an another)
4. Run sudo apt-get install /path/to/<keybase 5.8.1 deb file>
5. Set an upgrade hold on keybase to prevent re-upgrading to 5.9.0: sudo apt-mark hold keybase

[orgcontrib]

I didn't check the status on Windows as I moved my PC on a different (remote) location but there's an older case (#24364) for Linux. It's always better to use/update existing tickets (search through active/open issues before creating a new one). Remember that OSS projects are usually driven by unpaid volunteers. Imagine this was your project.

[paddyredbeard]

@orgcontrib re:

It's always better to use/update existing tickets (search through active/open issues before creating a new one).

Understood, however

1. The issue you linked did not appear in my searches,
2. These appeared to be emergent bugs in release 5.9.0 (I am a long-time user of KB on multiple platforms and have not previously seen these behaviors), and
3. Upon completing keybase log send, the output suggested creating a new issue in this queue that included the log ID submitted.

[orgcontrib]

Well, no need to explain. Take it as a friendly advice. I'm as well a long time user on multiple platforms.

[jacksongoode]

Getting the same issue with the mount point on Linux (Fedora 35) - the keybase mount service fails.

[gene1wood]

It manifested for me specifically with the mount issue on Ubuntu 18.04. I described it in #24764 along with some instructions on how to downgrade

[dr-kristau]

Same problem here on a fresh install of Ubuntu 21.10, downgrading to 5.8.1 as suggested works for me.

[sc6l6d3v]

Similar issue with AlmaLinux 8.5 (Centos 8 distro) with KB version: keybase version 5.9.0-20211217212642+29bfd9d39f

I can use my /run/user/... mountdir but any attempted traversal of the /keybase inode hangs that process. I had this on my old AlmaxLinux 8.5 system which I just upgraded to may have to bring that up again to compare what changed.

[StripedMonkey]

I've discovered two things that may be apart of this Firstly, calling /bin/ls / does work. It is only when color is enabled this is an issue. Secondly It seems that ls and any other file browser will call make a statx syscall to figure out the file type and fails due to some issue. This system call hangs forever. Running strace reveals that much.

Running /bin/ls instead of ls (which is aliased to --color) works fine.

[StripedMonkey]

In my humble opinion this isn't even strictly an issue with keybase, at least on the Linux side of things. System/lib calls like this should at least be timing out, not hanging forever.

[paddyredbeard]

I've discovered two things that may be apart of this Firstly, calling /bin/ls / does work. It is only when color is enabled this is an issue. Secondly It seems that ls and any other file browser will call make a statx syscall to figure out the file type and fails due to some issue. This system call hangs forever. Running strace reveals that much.

Running /bin/ls instead of ls (which is aliased to --color) works fine.

Although I've seen other issue reports mentioning problems when trying to ls the root filesystem path, these aren't problem behaviors I've experienced.

[sc6l6d3v]

Confirmed - easy to unalias ls, but why does it hang when trying: 'df -k' when 'df -k /' works just fine?

[StripedMonkey]

why does it hang when trying: 'df -k' when 'df -k /' works just fine?

I cannot replicate this issue 100% of the time so I cannot test until the bug appears again but I assume the issue lies in whatever system calls are made. Something about the keybase filesystem is hanging system calls that try and examine some particular aspect of it, but not all.

However it seems like this might not be the same issue as OP had, considering (IMO anyways) this is pretty operating system specific, and I have no idea how long this bug might have actually been around.

@paddyredbeard do you know what issues in particular are related to the issue I've described?

[sc6l6d3v]

why does it hang when trying: 'df -k' when 'df -k /' works just fine?

I cannot replicate this issue 100% of the time so I cannot test until the bug appears again but I assume the issue lies in whatever system calls are made. Something about the keybase filesystem is hanging system calls that try and examine some particular aspect of it, but not all.

However it seems like this might not be the same issue as OP had, considering (IMO anyways) this is pretty operating system specific, and I have no idea how long this bug might have actually been around.

@paddyredbeard do you know what issues in particular are related to the issue I've described?

The most annoying issue and one that I have never figured out is that the hang also hangs the strace trying to check it.

[StripedMonkey]

It hangs on the syscall that actually hangs. That's how/why I know that statx is at least one system call that will hang. You can call strace and look at the one its hung on.

[paddyredbeard]

@paddyredbeard do you know what issues in particular are related to the issue I've described?

I've not had trouble with df nor with ls or other standard OS calls. The original issue I had in Linux was on both Pop_OS! 20.04 LTS (based on Ubuntu 20.04 LTS) as well as on Ubuntu Budgie 20.04 LTS. On these systems kbfs was not properly mounted at /keybase and so kbfs files could not be accessed via that path (although they could be accessed at /run/user/<uid>/keybase/kbfs).

FWIW, I just now tried re-upgrading to 5.9.0 to reproduce the exact error output when trying to view the folders under /keybase, and curiously everything is working now (at least on the Pop_OS! system). I also do not have any trouble with ls, df, etc, as has been mentioned by others in this thread. I've installed a number of other OS package updates since first downgrading to keybase 5.8.1, so perhaps the problem behavior I've seen was due to a conflict with an earlier version of one of those? Hard saying not knowing.

PS. I also continue to see kbfs failing to mount to the K: drive on Windows 11 (as well as the GUI not launching). It's this mounting issue on both Windows and Linux that made me think there could be a broader problem with kbfs in the 5.9.0 release of the client.

[sc6l6d3v]

@paddyredbeard do you know what issues in particular are related to the issue I've described?

I've not had trouble with df nor with ls or other standard OS calls. The original issue I had in linux was on both Pop_OS! 20.04 LTS (based on Ubuntu 20.04 LTS) as well as on Ubuntu Budgie 20.04 LTS. On these systems kbfs was not properly mounted at /keybase and so kbfs files could not be accessed via that path (although they could be accessed at /run/user/<uid>/keybase/kbfs).

FWIW, I just now tried re-upgrading to 5.9.0 to reproduce the exact error output when trying to view the folders under /keybase, and curiously everything is working now (at least on the Pop_OS! system). I also do not have any trouble with ls, df, etc, as has been mentioned by others in this thread. I've installed a number of other OS package updates since first downgrading to keybase 5.8.1, so perhaps the problem behavior I've see was due to a conflict with an earlier version of one of those? Hard saying not knowing.

The main issue I see if that even with a raw ls I still cannot see anything from the redirector mount of /keybase despite it saying it is there without hanging:

keybase-redirector on /keybase type fuse (ro,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) /dev/fuse on /run/user/3675/keybase/kbfs type fuse (rw,nosuid,nodev,relatime,user_id=3675,group_id=1040)

[sc6l6d3v]

@paddyredbeard do you know what issues in particular are related to the issue I've described?

I've not had trouble with df nor with ls or other standard OS calls. The original issue I had in Linux was on both Pop_OS! 20.04 LTS (based on Ubuntu 20.04 LTS) as well as on Ubuntu Budgie 20.04 LTS. On these systems kbfs was not properly mounted at /keybase and so kbfs files could not be accessed via that path (although they could be accessed at /run/user/<uid>/keybase/kbfs).

FWIW, I just now tried re-upgrading to 5.9.0 to reproduce the exact error output when trying to view the folders under /keybase, and curiously everything is working now (at least on the Pop_OS! system). I also do not have any trouble with ls, df, etc, as has been mentioned by others in this thread. I've installed a number of other OS package updates since first downgrading to keybase 5.8.1, so perhaps the problem behavior I've seen was due to a conflict with an earlier version of one of those? Hard saying not knowing.

PS. I also continue to see kbfs failing to mount to the K: drive on Windows 11 (as well as the GUI not launching). It's this mounting issue on both Windows and Linux that made me think there could be a broader problem with kbfs in the 5.9.0 release of the client.

The redirector services seems fine:

keybase-redirector.service - Keybase Root Redirector for KBFS Loaded: loaded (/usr/lib/systemd/user/keybase-redirector.service; enabled; vendor preset: enabled) Active: inactive (dead) since Mon 2022-01-03 14:57:04 EST; 1h 18min ago Process: 513477 ExecStart=/usr/bin/keybase-redirector /keybase (code=exited, status=0/SUCCESS) Process: 513436 ExecStartPre=/usr/bin/keybase --use-root-config-file config get --direct --assert-false --assert-ok-on-nil disable-root-redirector (code=exited, status=0/SUCC> Main PID: 513477 (code=exited, status=0/SUCCESS)

[cmstrickland]

on my linux clients, it seems to me to be a problem with the redirector, which leads to the /keybase mount there in a blocking state. It gets wedged shutting down - killing the redirector service allows a blocked run_keybase task to complete and that seems to be the cause of the startup hangs. The /keybase mount is then not operable, but the application just detects this and points files to the appropriate user mount under /run/user . Occassionally I've managed to get this mount point wedged to the extent that it becomes blocked in the kernel during reads (i.e. process state D) and I imagine something like that could be the cause of occasional reports of filesystems and shells hanging while looking at /. Wedged mounts are tricky to resolve from user space, although a reboot will reliably clear up the stale state ;-)

it is possible to disable the redirector completely as you can see in the book https://book.keybase.io/guides/linux#configuring-kbfs

shutdown keybase, and make sure nothing is using kbfs sudo umount /keybase to clear the mount state sudo rmdir /keybase sudo keybase --use-root-config-file ctl redirector --disable

now run_keybase should do its thing. the only missing functioality will be /keybase/ paths as a magical entry point into the current users kbfs namespace. you can still access kbfs from the app or from the user mount under /run/user/<uid>/keybase/kbfs

this might be preferable to not upgrading the keybase client. If I get some time I will try and debug what's going on with the redirector in more detail, but this workaround unblocked my keybase use for now.

[heronhaye]

For anyone in this situation, does the following help?

sudo fusermount -uz /keybase
run_keybase

If so, I believe we have located the issue and should be able to push out a fix shortly.

[sc6l6d3v]

On AlmaxLinux 8.5 (Arctic Sphynx), if I do that the /keybase mount goes away.

Attempts to restart it:

systemctl --user restart keybase kbfs keybase-redirector

yield this:

% /bin/ls -l /keybase /bin/ls: cannot access '/keybase': Transport endpoint is not connected

It is fine under /run/user/NNNN/keybase/kbfs

I run a continuous top in a shell and on a new server sense the hanging on the /keybase mount as the load avg is around 2.00 when it should be closer to 0.00.

Anecdotally, I had to reboot the other day, and magically, the mount for /keybase worked fine.

This went away after another H/W update to SSDs and reboot.

[paddyredbeard]

For anyone in this situation, does the following help?

sudo fusermount -uz /keybase
run_keybase

If so, I believe we have located the issue and should be able to push out a fix shortly.

@heronhaye Yes, this seems to address the kbfs issues on a system runing Ubuntu Budgie 20.04LTS (though, as I suspect you know, needs to be re-executed for each session). Thank you! Looking forward to the update.

[heronhaye]

Hi, we believe we have fixed this issue for Linux. Please try upgrading Keybase. Thank you. You will likely have to reboot your computer after the upgrade.

[paddyredbeard]

Hi, we believe we have fixed this issue for Linux. Please try upgrading Keybase. Thank you. You will likely have to reboot your computer after the upgrade.

Thank you! Updating to 5.9.0-20220120174718.95a3939b3a does appear to have resolved the problems mounting kbfs.

[sc6l6d3v]

No joy here on AlmaLinux release 8.5 (Arctic Sphynx) after upgrading and rebooting:

% keybase version Client: 5.9.0-20220120174718+95a3939b3a Service: 5.9.0-20220120174718+95a3939b3a

still hangs on % /bin/ls /keybase

which is mounted:

% mount | grep keybase keybase-redirector on /keybase type fuse (ro,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) /dev/fuse on /run/user/3675/keybase/kbfs type fuse (rw,nosuid,nodev,relatime,user_id=3675,group_id=1040) /dev/sdb1 on /opt/keybase type ext4 (rw,relatime)

any details I can still share for Alma/Centos 8?

[heronhaye]

@sc6l6d3v Did you reboot your computer and then run run_keybase?

If so, can you run keybase log send and post the log id? Thanks.

[sc6l6d3v]

@heronhaye Did reboot and running run_keybase now. Will send logs shortly.

[sc6l6d3v]

@heronhaye Here's the link:

https://github.com/keybase/client/issues/new?body=[write%20something%20useful%20and%20descriptive%20here]%0A%0Amy%20log%20id:%209e6b3c89a7fed8afc7d9df1c

run_keybase hung but eventually came back after several minutes

I have it set up so that it runs from the service, btw:

% systemctl --user -l status keybase kbfs keybase-redirector hkatz@verbiet[1039 15:41:19]: systemctl --user -l status keybase kbfs keybase-redirector ● keybase.service - Keybase core service Loaded: loaded (/usr/lib/systemd/user/keybase.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-01-20 15:38:14 EST; 3min 7s ago Main PID: 19873 (keybase) CGroup: /user.slice/user-3675.slice/user@3675.service/keybase.service ├─12090 [keybase] └─19873 /usr/bin/keybase --use-default-log-file --debug service

Jan 20 15:38:13 verbiet systemd[10914]: Starting Keybase core service... Jan 20 15:38:14 verbiet systemd[10914]: Started Keybase core service.

● kbfs.service - Keybase Filesystem service Loaded: loaded (/usr/lib/systemd/user/kbfs.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-01-20 15:38:14 EST; 3min 7s ago Process: 19898 ExecStartPre=/bin/sh -c fusermount -uz "$(keybase config get -d -b mountdir)" (code=exited, status=1/FAILURE) Main PID: 19920 (kbfsfuse) CGroup: /user.slice/user-3675.slice/user@3675.service/kbfs.service └─19920 /usr/bin/kbfsfuse -debug -log-to-file

Jan 20 15:38:14 verbiet systemd[10914]: Starting Keybase Filesystem service... Jan 20 15:38:14 verbiet systemd[10914]: Started Keybase Filesystem service.

● keybase-redirector.service - Keybase Root Redirector for KBFS Loaded: loaded (/usr/lib/systemd/user/keybase-redirector.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-01-20 15:05:30 EST; 35min ago Main PID: 11009 (keybase-redirec) CGroup: /user.slice/user-3675.slice/user@3675.service/keybase-redirector.service └─11009 /usr/bin/keybase-redirector /keybase

Jan 20 15:05:30 verbiet systemd[10914]: Starting Keybase Root Redirector for KBFS... Jan 20 15:05:30 verbiet systemd[10914]: Started Keybase Root Redirector for KBFS.

[dr-kristau]

In Ubuntu 21.10, I upgraded and rebooted. I still have the problem, and have reported it with

my log id: 1cfdb4151888631e5f05901c

And $ journalctl -g keybase:

-- Boot 78227c64aecb49c7af117afbd61df5b3 --
Jan 21 20:20:07 neptune systemd[1499]: Starting Keybase core service...
Jan 21 20:20:17 neptune systemd[1499]: Started Keybase core service.
Jan 21 20:20:17 neptune systemd[1499]: Starting Keybase Filesystem service...
Jan 21 20:20:17 neptune systemd[1499]: Starting Keybase Root Redirector for KBFS...
Jan 21 20:20:17 neptune keybase[2297]: 2022-01-21T20:20:17.391905+01:00 ▶ [WARN keybase json.go:81] 001 Permission denied opening>
Jan 21 20:20:17 neptune keybase[2297]: 2022-01-21T20:20:17.391954+01:00 ▶ [WARN keybase json.go:81] 002 Permission denied opening>
Jan 21 20:20:17 neptune keybase[2297]: 2022-01-21T20:20:17.391979+01:00 ▶ [WARN keybase json.go:81] 003 Permission denied opening>
Jan 21 20:20:17 neptune sh[2320]: fusermount: entry for /home/tofol/keybase not found in /etc/mtab
Jan 21 20:20:17 neptune systemd[1499]: Started Keybase Root Redirector for KBFS.
Jan 21 20:20:57 neptune systemd[1499]: Started Keybase Filesystem service.
Jan 21 20:20:57 neptune systemd[1499]: Starting Keybase GUI...
Jan 21 20:20:57 neptune systemd[1499]: Started Keybase GUI.
Jan 21 20:20:57 neptune keybase_autostart.desktop[1897]: run_keybase: Success!
Jan 21 20:20:57 neptune systemd[1499]: app-gnome-keybase_autostart-1897.scope: Deactivated successfully.
Jan 21 20:21:18 neptune dbus-daemon[735]: [system] Activating via systemd: service name='org.bluez' unit='dbus-org.bluez.service'>
Jan 21 20:23:31 neptune kernel: INFO: task keybase:2335 blocked for more than 120 seconds.
Jan 21 20:23:31 neptune kernel: task:keybase         state:D stack:    0 pid: 2335 ppid:  1499 flags:0x00000000
Jan 21 20:25:32 neptune kernel: INFO: task keybase:2335 blocked for more than 241 seconds.
Jan 21 20:25:32 neptune kernel: task:keybase         state:D stack:    0 pid: 2335 ppid:  1499 flags:0x00000000
Jan 21 20:27:33 neptune kernel: INFO: task keybase:2335 blocked for more than 362 seconds.
Jan 21 20:27:33 neptune kernel: task:keybase         state:D stack:    0 pid: 2335 ppid:  1499 flags:0x00000000

I'm going to have to revert to the working version again.

[dr-kristau]

When I downgrade to keybase version 5.8.1-20210930160723+fefa22edc1, journalctl gives me identical output after reboot, except for the blocked tasks reported by the kernel at the end. Not many clues here I'm afraid, but everything works again, in the terminal I can navigate to the root of the file system, run ls and it won't hang.

[sc6l6d3v]

When I downgrade to keybase version 5.8.1-20210930160723+fefa22edc1, journalctl gives me identical output after reboot, except for the blocked tasks reported by the kernel at the end. Not many clues here I'm afraid, but everything works again, in the terminal I can navigate to the root of the file system, run ls and it won't hang.

Where is the repo to find this old version or did you do this solely through the platform installer as a downgrade?

[gene1wood]

You can find the amd64 build of that version at : https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/keybase_5.8.1-20210930160723.fefa22edc1_amd64.deb

[paddyredbeard]

When I downgrade to keybase version 5.8.1-20210930160723+fefa22edc1, journalctl gives me identical output after reboot, except for the blocked tasks reported by the kernel at the end. Not many clues here I'm afraid, but everything works again, in the terminal I can navigate to the root of the file system, run ls and it won't hang.

Where is the repo to find this old version or did you do this solely through the platform installer as a downgrade?

https://github.com/keybase/client/issues/24749#issuecomment-998906626

[sc6l6d3v]

You can find the amd64 build of that version at : https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/keybase_5.8.1-20210930160723.fefa22edc1_amd64.deb

Gene, Thanks for the link but I need a centos build or an rpm file. I tried a few permutations with .rpm and/or centos on the url above without success.

[sc6l6d3v]

Note: For debian-based installations, here is the workaround to downgrade, for those who may be interested...

1. Go to https://prerelease.keybase.io/ and then click the linux/deb link
2. Download the version 5.8.1 .deb file appropriate for your system (ie. the i386.deb file for 32-bit systems or the amd64.deb for 64-bit systems). Optionally (preferably) also download the corresponding PGP signature and verify the downloaded .deb file
3. Run sudo apt purge keybase to remove version 5.9.0. You may need to reboot at this point (on one system I needed to do so to cleanly install 5.8.1 but not an another)
4. Run sudo apt-get install /path/to/<keybase 5.8.1 deb file>
5. Set an upgrade hold on keybase to prevent re-upgrading to 5.9.0: sudo apt-mark hold keybase

Thanks - found the pre-release for rpm here.

[heronhaye]

Sorry folks, continuing to investigate.

[heronhaye]

@Dr-Kristau @sc6l6d3v If you get a chance, would you mind installing the latest Keybase linux version, disabling the redirector with sudo keybase --use-root-config-file ctl redirector --disable. After this and rebooting, is your system and Keybase functional again, with the exception of the /keybase mountpoint, which should be alternately accessible from $XDG_RUNTIME_DIR/keybase/kbfs?

This will help us narrow down the issue to the redirector as I have not been able to reproduce it locally.

[dr-kristau]

@heronhaye of course. Good news - the problem is resolved on my machine. I can access the root of the file system with ls, and my custom mountpoint (keybase config get mountdir) is working as expected.

[sc6l6d3v]

@Dr-Kristau @sc6l6d3v If you get a chance, would you mind installing the latest Keybase linux version, disabling the redirector with sudo keybase --use-root-config-file ctl redirector --disable. After this and rebooting, is your system and Keybase functional again, with the exception of the /keybase mountpoint, which should be alternately accessible from $XDG_RUNTIME_DIR/keybase/kbfs?

This will help us narrow down the issue to the redirector as I have not been able to reproduce it locally.

@heronhaye This was also confirmed on alma linux 8.5. I can access it on $XDG... but not on /keybase mountpoint. What's your guestimate to restore the root level?

[tgockel]

It seems like I have this issue -- submitted log ID 40c759c60814cee06b89aa1c.

I attempted workaround from @heronhaye earlier, but:

$> sudo fusermount -uz /keybase
$> run_keybase
Shutting down Keybase GUI...
Unmounting /run/user/1000/keybase/kbfs...
Unmounting and shutting down kbfsfuse...
Shutting down keybase service...
Starting via systemd...

This step hangs indefinitely.

• OS: Ubuntu 20.04.3 LTS
• Keybase Client: 5.9.0-20220120174718+95a3939b3a
• Keybase Service: 5.9.0-20220120174718+95a3939b3a

[karlvlam]

I have the same issue on my ubuntu 20.04 too. The keybase version is 5.9.0

[heronhaye]

@karlvlam, @tgockel, @sc6l6d3v, and all others having issues, please try out the latest 5.9.3 test build:

.deb: https://s3.amazonaws.com/tests.keybase.io/linux_binaries/deb/index.html .rpm: https://s3.amazonaws.com/tests.keybase.io/linux_binaries/rpm/index.html

A restart might be required before/after installing.

You can verify the signature on the binary against our code signing key at https://book.keybase.io/docs/server/our-code-signing-key.

I'm hoping this will resolve the issues, especially with ls / hanging. If you are still having issues, let me know. Otherwise I will promote this test build to a release in a few days. Thanks for your patience.

[heronhaye]

Thanks for your patience. On Linux, please upgrade to 5.9.3 via your package manager or https://keybase.io/docs/the_app/install_linux and let me know if you still run into any issues.

[tgockel]

After upgrading to 5.9.3, I have not experienced a hang. Thanks for the fix.