level-memview plays messages out of order on catchup sync

[pfrazee]

what i'm seeing is that order is not preserved within the same feed. in particular, replies are being processed before their parents. that causes the index to get screwy

[dominictarr]

Are they in the received order as per the timestamp on the log stream? there is not actually anything that per se that enforces the causal order

[pfrazee]

dunno, maybe i need to have a more savvy use of it. right now it's

var awaitSync = memview(sbot.ssb, processor, function () { return null })

but maybe it should be using the log sublevel?

[dominictarr]

what does processor look like? and why is the get method just null?

[pfrazee]

processor is https://github.com/ssbc/phoenix-api/blob/master/processor.js#L199-L208

get is null because it wasnt really necessary. i just need the getter functions to buffer until onSync

[dominictarr]

Right, but that is what the getter function do. what are you doing instead of using the getter functions?

[pfrazee]

i use the returned function as a sync guard, eg https://github.com/ssbc/phoenix-api/blob/master/index.js#L101-L133. awaitSync is the fn returned by memview (no getter behavior given, as i showed in the past comment)

i just need the getter functions to buffer until onSync

to make sure there's no semantic confusion -- two getters: the rpc api's getters, and the memview getter. the memview getter gives me the buffering, but there's no point in putting any state-fetching functionality into that func. the rpc api getters already have access to the state object, and, when we switch to disk-storage of computed state, they'll have access to the db then too

[pfrazee]

giving the log sublevel to memview doesnt work, of course, because log stores keys (not the messages themselves). getting the message introduces async, so then new messages (after the initial catchup) dont get synced in time for getters

replacing memview with live createLogStream and my own onSync guard doesnt work either, for the same reason - createLogStream has to get the messages, which is async

[pfrazee]

https://github.com/ssbc/phoenix-api/pull/10 appears to solve it fully

[dominictarr]

Ah, so I think what we can do to fix this, is to make a special log module that indexes by timestamp, but when reading back out it doesn't use the lookup for real time messages. This would have the advantages of both techniques.

[pfrazee]

yeah i bet that would work