search

ssbc / async-append-only-log

A new append-only-log for SSB purposes
16 stars 6 forks source link

Improve performance of isBufferZero #66

Open staltz opened 2 years ago

staltz commented 2 years ago

I had an idea to improve the performance of this.

If we end up "merging" deleted records, then we'll have huge records, and it'll be slow to check whether all the next (suppose) 50MB are zero bytes.

Instead what we can do is pass in an option to customize isBufferZero, similar to how we have opts.validateRecord that is passed in. This way, ssb-db2 could pass in a custom isBufferZero which just checks the first 10 bytes whether they are zero or not, because I suspect we never have an ssb-db2 record in the log where the first 10 bytes are zero and the next bytes after that are non zero.

staltz commented 2 years ago

@arj03 What do you think about this idea?

arj03 commented 2 years ago

I'm not so sure this is needed. You still have the block boundaries and the current implementation isBufferZero is already pretty optimized.