search

sourcenetwork / defradb

DefraDB is a Peer-to-Peer Edge Database. It's the core data storage system for the Source Network Ecosystem, built with IPLD, LibP2P, CRDTs, and Semantic open web properties.
447 stars 46 forks source link

Stabilize benchmark results #324

Open jsimnz opened 2 years ago

jsimnz commented 2 years ago

At the moment benchmarks are inconsistent in their results despite no notable code changes.

The entire goal of the benchmarks is to have a reliable metric to track the relative perf changes over time. Which is why they are run on a dedicated "clean" ec2 instance for each individual run.

AndrewSisley commented 2 years ago

For future reference - I had a play around with this in https://github.com/sourcenetwork/defradb/pull/359 taking the run-count all the way up to 40 and there was still significant differences between runs. I do think the host machine is too unstable for our benchmarks and we should look into sorting that out.