Open thlorenz opened 10 years ago
I'm still confused how this was happing, I want to get to the bottom of this!
Where you doing liveStream.install
in the client?
No, was doing that on the server. If you want to wade through the now fixed code, here is the server side and client side.
As I said, it is fixed now, before I just had liveStream.install(db)
server side and db.liveStream().on('data', ...)
client side.
Hope that helps. I was basically using it wrong, but if it is very important to you I can try to repro this with a smaller script tomorrow.
I'm mostly aiming to debug the documentation, or make it fail with a more obvious error message.
A reproduce script would be appreciated
As this script shows, this is not an isolated level-live-stream
problem, but most likely due to some deserialization going on in muxdemux
when used with multilevlel.
@juliangruber should I take this issue over there?
@thlorenz your script doesn't use multilevel, shouldn't it rather be a sublevel or level-live-stream problem then?
@juliangruber yes it's not using multilevel, that is why it doesn't repro the problem. Which is what led us to believe that this is not an isolated live-stream/sublevel problem. If you look at the stack trace above you'll see that my app blows up in json-buffer
triggered by mux-demux
.
It may not be too important since I can work around this by subscribing to the live stream on the actual sublevels instead the root db.
My question was just if we should move this issue to multilevel and I'll try to get around creating a script that isolates this issue as much as possible (it would use multilevel).
ah, ok I thought your script reproduced the problem. Sounds good to move over then.
hey guys here is a reduced test case.
https://github.com/soldair/level-live-stream_issues_7
https://github.com/soldair/level-live-stream_issues_7/blob/master/test/test.js
This is absolutely a bug that must be fixed. The client should never be allowed to take down the database server! Also just because you have sublevels doesn't mean every consuming client has to have logic using the fact that they exist.
the issue at hand is where to fix it!
the super hack i have to run now is
var through = require('through');
module.exports = function fix(db) {
var oldLiveStream = db.liveStream;
if(!oldLiveStream) return;
db.liveStream = function(){
return oldLiveStream.apply(this,arguments).pipe(through(function(data){
delete data.prefix;
this.queue(data);
}))
};
Object.keys(db.sublevels).forEach(function(k){
fix(db.sublevels[k]);
});
}
i'm real interested in getting a pull request in for this suggestions on the correct place to patch would help a lot.
deleting the prefix is just a hack if we don't actually know what is causing the problem. I think this is something to do with level-post, which basically adds post hook functionality a second time, since you are not exposing the sublevel the first time.
aha, I think the right fix is to have level-hooks compact the prefix to a string (which is also supported) but leave it there.
i can make that pr now! thanks for the advice like i said i want to remove my super hack asap
okay this is fixed - reinstall level-sublevel (to get level-hooks@4.4.5) and it should work.
can you make a PR that adds this as a test to level-sublevel?
all it needs to test for is that post hooks are always jsonifiable, even when using prehooks like this. Oh, by the way, you can also do: prefix: subdb.prefix()
and that will work even with the old code.
thanks! i just saw that you supported it and it was just too nice not to use
it's best to use .prefix()
since that works when you do batches even from the multilevel client.
Let's say on the server you have something like:
Consuming on the client side like this:
Since we did this incorrectly (i.e. attached live stream to root level instead of to the sublevels separately) we'll get a stack overflow error in 'json-buffer' due to the
_parent
on the sublevels which causes a circular reference:The solution here is to document to not do that and/or detect this earlier and throw a more meaningful error.