staltz
commented
3 years ago Allow customizing when should
decryptrun insidelog.stream
I tested this idea, running a db.query() without decrypt takes 350ms (first JITDB run, i.e. creates the core indexes and bitvector index from scratch). With decrypt it takes 1300ms. That's 3.7x heavier.
I have an idea that is basically just pass down a boolean decrypt: true in the operators, and then JITDB will call log.stream({ decrypt: true }) if it detects that any index in the query had decrypt: true. E.g. db.query(and(type('post', {decrypt: false}), etc) basically will pass the opts down to JITDB and JITDB will pass those opts back to ssb-db2/log.js, which then will not run decrypt.
This way we let the query developer choose wisely when to do decryption or not. The default (is it false or true) can be discussed.
cc @arj03
arj03
Not really an issue, I just want some place where I can report my analysis on the benchmark runs.
On my computer
npm run benchmark-no-createfrom #107 reported8030msfor theinitial indexingtest.In a profiler, the big picture looks like this:
There are roughly 4 parts there:
requireeverything, set up secret-stack plugins, all that stuff (looks like spiky chaos on the left)createMissingIndexes, and in this process it needs to do alog.stream, and thislog.streamperformsdecryptand so forth, and then the final spike before the silence isgetBitsetForOperationetcStats
Ideas
Separate benchmark reports for migration
Migration's duration should be measured separately from base indexing.
Don't strictly require indexing base index before db.query()
If db.query() would only use JITDB, we shouldn't block it from running until base index is done. In the example in
benchmark/index.js, we don't use the base index, so we should be getting the db.query results much earlier.Allow customizing when should
decryptrun insidelog.streamFor the
db.queryin this example, we didn't need to consider private messages. There should be a way of specifying that we don't needdecryptinsidelog.streamwhen indexing something in JITDB. In the ideal situation, JITDB would automatically detect whether it needs unboxing, and pass that down as a parameter tolog.stream, but a manual opt-in might also be okay.Push-streams could/should be paused
If we're doing asynchronous work upon receiving an item from a push-stream, we should pause the push-stream until the async work is done, and then resume it. This would use the backpressure capabilities of push-stream. Probably we're not in deep trouble if we don't do this, but depending on the corner case, if we don't do backpressure, we might just accumulate or exhaust the computer's RAM of tasks and incoming data, and this can have adverse effects elsewhere.
cc @arj03