search

ssbc / ssb-ebt

secure scuttlebutt replication with epidemic-broadcast-trees
MIT License
18 stars 10 forks source link

Butt2 #67

Closed arj03 closed 2 years ago

arj03 commented 2 years ago

Almost ready. Depends on https://github.com/ssbc/ssb-db2/pull/343

arj03 commented 2 years ago 
 Classic:
  - large: after 3000 ms bob has 5370 msgs
  - small: after 3000 ms bob has 8085 msgs
  - using ssb-validate2
  - disables batch validation (debouncer)

  Butt2:
  - large: after 3000 ms bob has 7057 msgs
  - small: after 3000 ms bob has 7100 msgs

butt2 is much more stable as it doesn't matter what the content size is (we never encode/deocde) this. Note this is on the same machine and not super optimized. Sending is a bit slower, so if you would only measure the receiving side, then the numbers would be better for butt2. And with batch validation butt2 is around 9800 msgs after 3 sec.

arj03 commented 2 years ago

With the latest changes butt2 is now at around 7500-8000 messages. So basically as good as classic on small messages and a lot faster on larger messages.

arj03 commented 2 years ago

And update on this @staltz. I put in the formats-split branch and made some minor modifications to get things running. Both in this module and in butt2, and both rather minimal. That enabled me to test both classic and buttwoo speed. Classic is roughly the same as before: large msgs 5200 msgs / 3 secs, small msgs 8800 msgs / 3 secs. Buttwoo is now 6800 for both. Was 7800 before. So not super bad, I havn't yet looked into why it is slower. Wanted to let you know how things looks. Also this is a good test that things are working :) I found some minor things as you can see.

staltz commented 2 years ago

@arj03 That's good to hear. I was expecting some performance regression, because there are probably some double work in some places, like with getFeedId or fromNativeMsg. I have an idea to use WeakMap that should essentially cache the results of these functions instead of calling them again.

By the way, I don't know how you integrated ssb-db2 formats-split here, but you should use onMsgAdded instead of post (you may have noticed there is no buttPost).

WeakMap didn't work in bipf, because there we had nested WeakMaps, so the creation of a WeakMap turned out to be a bottleneck. But WeakMaps worked extremely well in jitdb. I think it's the same case now for feed formats in ssb-db2 because we will create the map only once.