search

keybase / keybase-issues

A single repo for managing publicly recognized issues with the keybase client, installer, and website.
902 stars 37 forks source link

KBFS broken: Gap too large between event and global Merkle roots #3778

Open rcorre opened 4 years ago

rcorre commented 4 years ago

As of this morning, I can't connect to kbfs. keybase.kbfs.log says:

2019-12-31T09:03:45.164843-05:00 ▶ [DEBU kbfs(kbfsfuse) tlf.go:89] 20f Loading root directory for folder rcorre (type: private, filter error: true) [t
ags:FID=REDACTED]
2019-12-31T09:03:45.164952-05:00 ▶ [DEBU kbfs kbfs_ops.go:1087] 210 getMaybeCreateRootNode(/keybase/private/rcorre, , false) [tags:FID=LAzE5np2203iJ9T
29t747g]
2019-12-31T09:03:45.165256-05:00 ▶ [DEBU kbfs mdserver_remote.go:635] 211 Read revision 1926 for TLF REDACTED from the disk ca
che [tags:FID=REDACTED]
2019-12-31T09:03:45.188777-05:00 ▶ [DEBU kbfs mdserver_remote.go:670] 212 Read revision 1926 for TLF REDACTED from the server
[tags:FID=REDACTED]
2019-12-31T09:03:45.254222-05:00 ▶ [DEBU kbfs kbpki_client.go:203] 213 Revoked verifying key REDACTED
cfa1462bdbe0a for user a3ab07daf19ef189434de63ecf43c819 passes time check (revoked time: 2019-12-30 12:57:00 -0500 EST vs. server time 2019-12-30 12:5
2:27.527 -0500 EST, slack=1m0s) [tags:FID=REDACTED]
2019-12-31T09:03:45.254419-05:00 ▶ [DEBU kbfs kbfs_ops.go:1150] 214 Done: Gap too large between event and global Merkle roots: gap=-8h15m0s, timeToChe
ck=2019-12-30T12:57:00-05:00, latestRootTime=2019-12-30T23:35:33-05:00
github.com/keybase/client/go/kbfs/libkbfs.(*MDOpsStandard).checkMerkleTimes
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/md_ops.go:315
github.com/keybase/client/go/kbfs/libkbfs.(*MDOpsStandard).checkRevisionCameBeforeMerkle
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/md_ops.go:476
github.com/keybase/client/go/kbfs/libkbfs.(*MDOpsStandard).verifyKey
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/md_ops.go:591
github.com/keybase/client/go/kbfs/libkbfs.(*MDOpsStandard).processMetadata
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/md_ops.go:872
github.com/keybase/client/go/kbfs/libkbfs.(*MDOpsStandard).processMetadataWithID
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/md_ops.go:1052
github.com/keybase/client/go/kbfs/libkbfs.(*MDOpsStandard).processSignedMD
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/md_ops.go:1072
github.com/keybase/client/go/kbfs/libkbfs.(*MDOpsStandard).getForTLF
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/md_ops.go:1094
github.com/keybase/client/go/kbfs/libkbfs.(*MDOpsStandard).GetForTLF
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/md_ops.go:1101
github.com/keybase/client/go/kbfs/libkbfs.journalMDOps.getForTLF
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/journal_md_ops.go:289
github.com/keybase/client/go/kbfs/libkbfs.journalMDOps.GetForTLF
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/journal_md_ops.go:300
github.com/keybase/client/go/kbfs/libkbfs.(*KBFSOpsStandard).getOrInitializeNewMDMaster
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/kbfs_ops.go:952
github.com/keybase/client/go/kbfs/libkbfs.(*KBFSOpsStandard).getMaybeCreateRootNode
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/kbfs_ops.go:1147
github.com/keybase/client/go/kbfs/libkbfs.(*KBFSOpsStandard).GetRootNode
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libkbfs/kbfs_ops.go:1230
github.com/keybase/client/go/kbfs/libfuse.(*TLF).loadDirHelper
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libfuse/tlf.go:114
github.com/keybase/client/go/kbfs/libfuse.(*TLF).loadDirAllowNonexistent
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libfuse/tlf.go:154
github.com/keybase/client/go/kbfs/libfuse.(*TLF).Lookup
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/kbfs/libfuse/tlf.go:215
github.com/keybase/client/go/vendor/bazil.org/fuse/fs.(*Server).handleRequest
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/vendor/bazil.org/fuse/fs/serve.go:1073
github.com/keybase/client/go/vendor/bazil.org/fuse/fs.(*Server).serve
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/vendor/bazil.org/fuse/fs/serve.go:887
github.com/keybase/client/go/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
        /build/keybase/src/.gopath/src/github.com/keybase/client/go/vendor/bazil.org/fuse/fs/serve.go:434
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1357 [tags:FID=REDACTED]

The only thing I can think of different than normal is that I revoked a device yesterday (a different one from the device I'm seeing this on). I can still view chats, but can't access KBFS. When I go through the GUI, it says An unexplainable error has occurred.

heronhaye commented 4 years ago

Still broken?

rcorre commented 4 years ago

Still broken, same error (after a reboot)

rcorre commented 4 years ago

I'm encountering the same on my other device (a phone).

On the plus side, I've noticed there's now a syncing data for offline access option, which I intend to turn on as soon as I get access to my files again :)

rcorre commented 4 years ago

@heronhaye it doesn't look like this is something that will just resolve with waiting, as the two times it is checking, "revoked time" and "server time" are always the same (revoked time: 2019-12-30 12:57:00 -0500 EST vs. server time 2019-12-30 17:52:27.527)

tetrus93 commented 4 years ago

the same problem i have there at the moment the same 8 hours 15 minutes delay of somekind

rcorre commented 4 years ago

For anyone encountering this, @songgao reached out and mentioned there were some server-side fixes that were supposed to fix this but for some reason didn't for me. They're investigating further, but are understandably a bit short-staffed during the holidays.

songgao commented 4 years ago

@tetrus93 did you revoke a device lately? Could you send us a log please?

tetrus93 commented 4 years ago

did you revoke a device lately?

i think revoked a device on 30th december and the whole thing started go weird

looking at logs, it started somewhere around right after i revoked the device. kbfs_error.txt keybase.kbfs.log not sure if these logs can help, but i will try pick every log i could...

(for some reason, doing "keybase log send" results in timeout, error: (Client.Timeout exceeded while awaiting headers), saying this just in case)

tetrus93 commented 4 years ago

is there any updates about this?

edit: wait nvm, i see some updates about this: https://github.com/keybase/client/commit/c496a8adf0a7c8b5a55927fda793f4603d1c5d92

maxtaco commented 4 years ago

@tetrus93: @songgao can provide more info, but I think there is a fix coming in the next release.

songgao commented 4 years ago

Yes. Sorry for the delay. The next release should include the fix. If anyone here feel like trying a test build instead of waiting for our public release, please let me on Keybase and I'll find you a test build link.