keybase / kbfs

Keybase Filesystem (KBFS)
https://keybase.io/docs/kbfs
BSD 3-Clause "New" or "Revised" License
1.21k stars 137 forks source link

/keybase "resets" at random times #1700

Open kg4zow opened 6 years ago

kg4zow commented 6 years ago

I've noticed several occasions over the past few weeks where my /keybase mount seems to "reset". I'll be working with a file in a kbfs directory, and when I try to save it, the program I'm using will throw an error message. The message is different based on what program it is, but they all boil down to some variant of "directory not found". When I investigate what happened, it looks something like this...

(jms1@laptop) 13 /keybase/private/jms1/work $ ls
README.md
(jms1@laptop) 14 /keybase/private/jms1/work $ bbedit README.md
(jms1@laptop) 15 /keybase/private/jms1/work $ 

(work for a bit, save file, no problems) (go to lunch, come back, more edits, save file, BBEdit throws an error message)

(jms1@laptop) 15 /keybase/private/jms1/work $ ls
ls: .: No such file or directory
(jms1@laptop) 16 /keybase/private/jms1/work $ cd /keybase/private/jms1/work
(jms1@laptop) 17 /keybase/private/jms1/work $ ls
README.md
(jms1@laptop) 18 /keybase/private/jms1/work $

I more or less understand what happened (something caused the filesystem to un-mount and re-mount itself, resulting in all existing file handles becoming invalid) but it seems like it's been happening more and more often over the past few weeks.

Is this a known issue?

Currently using keybase version 2.3.0-20180710171051+64615d6258 on OSX 10.13.6, although it was happening with earlier versions as well (I'm pretty good about updating the software when the pop-up appears.)

strib commented 6 years ago

Updating the software while you're cd'd into a directory (which causes a mount/unmount of the file system) is usually what causes this. Though it could be something else too, like a crash in our code.

If you don't think it corresponds to updates, please do a keybase log send and put the resulting log ID here, and we'll take a look.

kg4zow commented 6 years ago

It just happened. This time I happened to catch the Keybase icon "blink" on the screen for a bit, and then the current directory in which I was working became invalid.

Log ID: 0e0e02372e0b80940b18141c

Also ran keybase --version after doing the keybase log send ... the version number did not change.

strib commented 6 years ago

Thanks. Looks like you are hitting keybase/client#12817. Can you please try the build referenced here and see if it helps?

kg4zow commented 6 years ago

Installed keybase version 2.4.0-20180719225238+67ef6ce70a, will follow up here in a day or two.

nwbt commented 6 years ago

Seeing the same issue with version 2.3.0-20180710171051+64615d6258 on macOS 10.13.6 (17G65)

strib commented 6 years ago

@nwbt above I linked to a build you can try.

nwbt commented 6 years ago

@strib seems to be working so far. I'll follow up in a few days. Also I'm seeing a constant state of "UPLOADING CHANGES..." though it does not seem to impact functionality.

screen shot 2018-07-26 at 11 16 05 screen shot 2018-07-26 at 11 14 23
strib commented 6 years ago

@nwbt If you do a keybase log send, and post the resulting ID here, I can take a look at what might be stuck uploading, if anything. Though note that the previous bug would have prevented uploading anything you had written before, so it might just be draining that all now.

nwbt commented 6 years ago

@strib thanks! still hanging on UPLOADING CHANGES... log ID 30ed30080ca8e2146474db1c let me know how I can help

strib commented 6 years ago

Hmm, I don't see any actual uploads happening anymore.

Can you paste in the output of cat /keybase/.kbfs_status here? (It contains local paths that expose your local username, so if you prefer, you can copy it into a private folder with me instead: cp /keybase/.kbfs_status /keybase/private/strib,nwbt/kbfs_status.)

nwbt commented 6 years ago

The /keybase/.kbfs_status file does not exist anywhere on the file system

strib commented 6 years ago

It's created when you try to read it -- it's a special virtual file that we make when you ask for it. Just try to cat or cp it and see.

nwbt commented 6 years ago

very cool ... shared it with you privately

strib commented 6 years ago

@nwbt ah I see now, one of your folders (a private one shared with another user) is stuck, unable to flush its data to our servers. It's stuck in what we call a "conflict" state, where it's trying to reconcile your local data with the current server-based state of the folder. But due to what looks like a bug in our code, it can't reconcile them, and is failing with this error:

2018-07-26T09:32:31.631406-07:00 ▶ [DEBU kbfs(CR 154983ba) conflict_resolver.go:3254] e68 Finished conflict resolution: No chain found for BlockPointer{ID: 0165391bab109fdb9c7e43f8149a6ab488303b172b3cb3a88a3f77dd8596db6a6a, KeyGen: 3, DataVer: 3, Context: Context{Creator: 879f106a3c4cfcc1e8603ec6550a5b19}, DirectType: direct} [tags:CRID=WTUXf9VnuesSU9sR8KEMCw]

This might be the same underlying problem as keybase/client#11373, which we haven't fixed yet.

Unfortunately, due to the original error you were having, any logs that would have helped us figure out the issue are probably long gone from your machine.

But, you can work around this and fix the problem with the following steps I think:

  1. Save any data from that shared folder, that you can't recover from anywhere else, onto your local hard drive by copying it out of KBFS. (If you don't know which shared private folder I'm talking about, write me on Keybase -- I don't want to mention the other user publicly. You have about 150 MB of unsynced data in the folder. You can probably also figure out the name from the .kbfs_status file you shared with me by looking at the unflushed file list.)
  2. Completely quit out of the Keybase app, so that the mounted file system is no longer available.
  3. Run this command from the terminal:
    mv "/Users/$USER/Library/Application Support/Keybase/kbfs_journal/v1/0120a83cf3fbe8bc669d07484d6c48f9e5dc-154983ba38adeea3" "/Users/$USER/Library/Application Support/Keybase/kbfs_journal/v1/154983ba38adeea3.bkp"
  4. Restart Keybase

Now you can copy all that data back into the shared folder and things should work again. Sorry for the inconvenience!

kg4zow commented 6 years ago

It seems to have helped "more or less", i.e. I've only noticed it happen once since installing the debug build (which is a big improvement over 3-4 times per hour).