live queries should be aware of deleted records

staltz commented 2 years ago

Context

There's a crash in Manyverse with this stack trace:

TypeError: Cannot read property '0' of null
    at Object.read [as decode] (/data/data/se.manyver/files/nodejs-project/index.js:95407:14)
    at Object.seekKey2 (/data/data/se.manyver/files/nodejs-project/index.js:48349:24)
    at isValueOk (/data/data/se.manyver/files/nodejs-project/index.js:34315:25)
    at /data/data/se.manyver/files/nodejs-project/index.js:34797:31
    at /data/data/se.manyver/files/nodejs-project/index.js:75597:23
    at /data/data/se.manyver/files/nodejs-project/index.js:72339:16
    at /data/data/se.manyver/files/nodejs-project/index.js:37966:7
    at /data/data/se.manyver/files/nodejs-project/index.js:90290:5
    at /data/data/se.manyver/files/nodejs-project/index.js:32394:9
    at /data/data/se.manyver/files/nodejs-project/index.js:90295:5

From top to bottom, that's:

fast-varint
bipf
jitdb
...

Problem

Live streams in jitdb are bumping into deleted records. I'm not sure why, but they are.

This is from Manyverse 0.2208.5 which was using jitdb@7.0.2, ssb-db2@4.2.1, and AAOL@4.3.6. This might have changed in the newest version of Manyverse that's using AAOL@4.3.7 which fixed live streams post-compaction.

Anyway, seems reasonable that we have to treat null records from live streams.

Solution

isValueOk returns false if the record is deleted.

github-actions[bot] commented 2 years ago

Benchmark results

Part	Speed	Heap Change	Samples
Count 1 big index (3rd run)	0.9ms ± 0.66ms	44.08 kB ± 31.02 kB	36
Create an index twice concurrently	778.46ms ± 6.83ms	24.91 kB ± 25.64 kB	69
Load core indexes	1.55ms ± 0.03ms	113.95 B ± 494.57 B	5662
Load two indexes concurrently	665.51ms ± 6.3ms	102.75 kB ± 312.3 kB	17
Paginate 10 results	19.94ms ± 0.14ms	-19.67 kB ± 41.81 kB	28
Paginate 20000 msgs with pageSize=5	7873.9ms ± 431.83ms	-1.07 MB ± 1.56 MB	5
Paginate 20000 msgs with pageSize=500	505.78ms ± 3.3ms	169.28 kB ± 820.47 kB	22
Query 1 big index (1st run)	760.51ms ± 4.42ms	12.26 kB ± 66.2 kB	71
Query 1 big index (2nd run)	394.84ms ± 3.87ms	26.35 kB ± 30.5 kB	35
Query 3 indexes (1st run)	868.93ms ± 6.18ms	-31.94 kB ± 58.96 kB	62
Query 3 indexes (2nd run)	269.82ms ± 1.59ms	28.42 kB ± 32.46 kB	48
Query a prefix map (1st run)	328.3ms ± 4.34ms	43.65 kB ± 160.5 kB	20
Query a prefix map (2nd run)	11.17ms ± 0.5ms	57.57 kB ± 85.49 kB	20

github-actions[bot] commented 2 years ago

Benchmark results

Part	Speed	Heap Change	Samples
Count 1 big index (3rd run)	0.91ms ± 0.71ms	24.81 kB ± 31.13 kB	38
Create an index twice concurrently	598.41ms ± 1.76ms	4.33 kB ± 25.05 kB	91
Load core indexes	1.27ms ± 0.03ms	98.27 B ± 312.49 B	6808
Load two indexes concurrently	695.37ms ± 14.92ms	56.31 kB ± 295.68 kB	17
Paginate 10 results	26.13ms ± 1.44ms	-27.7 kB ± 71.81 kB	22
Paginate 20000 msgs with pageSize=5	7100.35ms ± 226.32ms	3.34 MB ± 2.81 MB	5
Paginate 20000 msgs with pageSize=500	504.98ms ± 3.33ms	-137.56 kB ± 648.32 kB	21
Query 1 big index (1st run)	742.29ms ± 3.21ms	-29.36 kB ± 61.34 kB	73
Query 1 big index (2nd run)	399.2ms ± 4.49ms	11.44 kB ± 50.8 kB	35
Query 3 indexes (1st run)	852.77ms ± 3.1ms	31.23 kB ± 65.93 kB	63
Query 3 indexes (2nd run)	240.88ms ± 1.19ms	11.73 kB ± 34.05 kB	54
Query a prefix map (1st run)	344.2ms ± 6.15ms	-35.35 kB ± 728.77 kB	20
Query a prefix map (2nd run)	12.63ms ± 1.09ms	20.84 kB ± 178.75 kB	21

ssbc / jitdb