attestate / kiwistand

kiwistand is a p2p node client for a web3 writer friendly Hacker News that nobody controls but everybody co-owns
https://news.kiwistand.com
GNU General Public License v3.0
63 stars 26 forks source link

use a fast tree DFS to enumerate the leaves #84

Closed freeatnet closed 1 year ago

freeatnet commented 1 year ago

This PR implements a DFS trie walk as a generator.

Benefits:

Benchmarks on 1000 runs for old and new walker (not accounting for message ordering time):

[0] posts-oo4r3t-{"from":null,"amount":null,"startDateTime":null}-new way: 25.166s
[0] posts-oo4r3t-{"from":null,"amount":null,"startDateTime":null}-old way: 36.618s

[0] posts-i0b6ni-{"from":0,"amount":500,"startDateTime":null}-new way: 5.239s
[0] posts-i0b6ni-{"from":0,"amount":500,"startDateTime":null}-old way: 23.746s

[0] posts-2ym77v-{"from":1000,"amount":500,"startDateTime":null}-new way: 13.996s
[0] posts-2ym77v-{"from":1000,"amount":500,"startDateTime":null}-old way: 30.884s

[0] posts-8xq9vn-{"from":2000,"amount":500,"startDateTime":null}-new way: 23.039s
[0] posts-8xq9vn-{"from":2000,"amount":500,"startDateTime":null}-old way: 34.398s

I haven't had a chance to test startDateTime — and I think we can improve performance even more if we control traversal/select branches which we want to descend.

TimDaub commented 1 year ago

I haven't had a chance to test startDateTime — and I think we can improve performance even more if we control traversal/select branches which we want to descend.

true, but anyways how you implemented it seems to be conceptually compatible with minTimestamp, in that it is later applied.

TimDaub commented 1 year ago

Thank you very much, @freeatnet; this is excellently engineered! Very much appreciated