search

ssbc / ssb-server

The gossip and replication server for Secure Scuttlebutt - a distributed social network
1.69k stars 164 forks source link

"Key not found in database" replication error #269

Closed pfrazee closed 8 years ago

pfrazee commented 8 years ago

I began to notice missing-message notices in my threads, which indicates that data is not being replicated evenly through the network. Some of my peers are receiving updates, and replying to them, whereas I am not.

It's possible they're conversing with a FoaF which is outside of my network. But, I sshed into my pub and noticed this was frequently happening in the logs:

info @0GL REPL  start
  "@uRECWB4KIeKoNMis2UYWyB2aQPvWmS3OePQvBj2zClg=.ed25519"
info @0GL REPL  start
  "@im4Qn0fCzpD3YfsegHFLJzkNXYUb/nYnlfuCf+LmPuM=.ed25519"
@/RM1Id8j05uitIt6iwMpiivnCqHcbcC1IHyi5FrvLLQ=.ed25519 undefined [NotFoundError: Key not found in database]
info @0GL REPL  start
  "@/RM1Id8j05uitIt6iwMpiivnCqHcbcC1IHyi5FrvLLQ=.ed25519"
@hxGxqPrplLjRG2vtjQL87abX4QKqeLgCwQpS730nNwE=.ed25519 undefined [NotFoundError: Key not found in database]
info @0GL REPL  start
  "@hxGxqPrplLjRG2vtjQL87abX4QKqeLgCwQpS730nNwE=.ed25519"
@Ia0xWQGJSTjRfYjHDDAFizXR9e8l5RQctTqYcbtR+Es=.ed25519 undefined [NotFoundError: Key not found in database]
info @0GL REPL  start
  "@Ia0xWQGJSTjRfYjHDDAFizXR9e8l5RQctTqYcbtR+Es=.ed25519"
@ye+QM09iPcDJD6YvQYjoQc7sLF/IFhmNbEqgdzQo3lQ=.ed25519 undefined [NotFoundError: Key not found in database]

That error output is the hard evidence of a bug. Since I havent confirmed yet why I'm missing those messages, I'm only suspicious that my missing messages relate to these error dumps.

We need to find where this is occurring and output better diagnostic information. If there's a bug, we should solve it as well.

pfrazee commented 8 years ago

(This is occurring on the latest build.)

dominictarr commented 8 years ago

this happened to me last week. I could see it was me that was missing from one thread. but also johnny.

On Mon, Dec 21, 2015 at 5:00 AM, Paul Frazee notifications@github.com wrote:

(This is occurring on the latest build.)

— Reply to this email directly or view it on GitHub https://github.com/ssbc/scuttlebot/issues/269#issuecomment-166131409.

pfrazee commented 8 years ago

@dominictarr is it possible this has to do with recent changes to replication, maybe the changes to the output of latestSequence? Maybe old clients aren't working with it?

pfrazee commented 8 years ago

I wasnt able to find it on my pub, but I was able to reproduce this bug on my local device when I called getLatest. I was using sbot 7.3.4, but had previously used 7.4.3 (I accidentally downgraded in some testing) which makes me think the lastDB's updated schemas were causing the error for me. I'm still trying to locate why my pub is emitting the error.

pfrazee commented 8 years ago

slaps forehead this was a TOTAL MISDIRECTION. Here's the source of it: https://github.com/ssbc/scuttlebot/blob/master/plugins/invite.js#L43

Easy fix, remove the log and we're done.