steveyen
commented
9 years ago Also, slightly related to https://github.com/couchbaselabs/cbft/issues/50
Open steveyen opened 9 years ago
steveyen
commented
9 years ago Also, slightly related to https://github.com/couchbaselabs/cbft/issues/50
Capturing chat between Marty & Steve on this issue...
Perhaps somewhat related to https://github.com/blevesearch/bleve/issues/186
Right now if cbft sees an error from a bleve batch, cbft doesn't have a good error handling approach right now... just a hammer... shut down the pindex, and restart it...
Which might ultimately lead to a DCP rollback, and then a really inefficient infinite loop as a DCP rebuild probably leads to the same eventual error.
Similar issue with elasticsearch and xdcr.
Wondering if there's a way to handle this in a general way, like this is a doc-specific error... versus some "nope, this affects everything kind of error" (like out of disk space).
Everyone says, can't you just ignore indexing errors. The problem is its really easy to end up with indexes that are missing 1 or 2 docs and you'd never know it. If you have a million docs you're not going notice missing docs just by searching.
Perhaps need to put those off to the side somewhere on some exception queue/ring where the user can see those. Possible to store these using SetInternal. Or could force them to a different mapping doesn't store anything.
Sort of like the "stub" documents we store in xdcr for binary docs
Also, gonna need enough info from the batch error result to be able to tell which way to go.
So there are basically 2 kinds of errors...
1 - errors that we can attribute to a single document.
2 - and errors that happen after we've lost the ability to know which doc caused the problem.
Also, I haven't reviewed it, but i think if it happens after the mapping stage, you're going to be lacking info -- everything is commingled into kv rows at that point disassociated from the document that created them, or some rows which are essentially cross-document.