emit('sync') event lost over multilevel

[ahdinosaur]

hi, thanks for level-live-stream! :)

i'm using the 'sync' event to know when all the historical data has been read in order to emit that data in a chunk before emitting new chunks based on real-time changes.

assuming the 'sync' event is lost because anything that isn't part of the readable stream is lost, can there be a 'notifyOnSync' option that includes a single piece of data in between reading historical data and reading real-time data like

{
  type: 'sync'
}

?

i implemented the above option here. are these changes worthy of a pull request?

[dominictarr]

hmm, @juliangruber what do you think?

Maybe simpler: what if each data object had a {key, value, old: true} on it then you don't need an extra item. So db.liveStream().pipe(db.writeStream()) would still work.

[ahdinosaur]

great idea, embedding the old/new status in the (meta-)data makes much more sense.

EDIT: however, there would be no way to tell when the old was done before the new started if no new changes were happening.

[dominictarr]

That is an excellent point. Maybe instead we want lastOfOld: true that would be close to the behavior of on('sync',...)

[juliangruber]

lastOfOld would be nasty to implement as you have to wait if the stream emits the close event before sending out each chunk. This is the first case where a multilevel stream needs to emit custom events, right?

[dominictarr]

Ah, yes. good point... this would mean that either mux-demux needs to send custom events... or, alternatively... you could have the manifest include some javascript functions, (instead of just json) which could have code that rehydrated the stream, interpreting special messages such as the one with the sync event.

this would also be useful for stuff that has a streaming protocol, like scuttlebutt.

The simple way would just be to make a module that turns the sync event into a dataevent ({sync: true}) and then another thing that monkeypatches emit and checks for {sync: true} and emits a sync event in that case. (this is what the hydrate function would do, except this would have to be applied manually)

[juliangruber]

And the general solution a module that overrides EventEmitter.prototype.emit and passes all the non-standard stream events of the stream.

[dominictarr]

That would require modifying mux-demux, and I'd rather not encourage wacky streams like that (yes, even though I added this feature to live-stream :). The other alternative is for {sync: true} to just be a message in the stream...

@ahdinosaur can you describe your use-case? what problem are you using the sync event to solve?

[ahdinosaur]

my use case is in bacon.level, a module i made to integrate bacon.js with level, which to be honest i'm not sure if i will continue to develop for unrelated reasons.

before the sync event, i use the incoming data to fill an object containing the state of the level. when the sync event occurs, i send a bacon event with the entire object containing the initial state. on every subsequent data event, i send a bacon event with the object containing the modified state.

[dominictarr]

ah, so you don't need this anymore?

[ahdinosaur]

indeed. i think it is still an issue, but i'm fine with closing this until someone needs this.

thanks for your help!

[dominictarr]

lets just leave it open until then, so that it's easier for them to find this.