Benchmark quality

[mweidner037]

Bring benchmarks up to paper quality:

• [ ] Multiple trials with average and stddev
• [ ] Warmup trials
• [ ] Report memory using profiler instead of (unreliable) process.memoryUsage().heapUsed

[mweidner037]

• [ ] Read the text trace from a file instead of using a JSON import. For some reason, the latter causes much more memory usage, which makes heap profiles take longer.

[mweidner037]

• [ ] Rotate replicaIDs over time, for realism
• [ ] Use a prng to generate fixed random values
• [ ] Check final state after loading

[scottmessinger]

It would be great to see the results side by side with the results from other libraries. It seems this library is competitive in regards to speed at cost of storage space. If i’m reading the results correctly, the final document is around 800KB compared to 250KB with automerge or others. Is that right?

[mweidner037]

@scottmessinger I made a PR on the crdt-benchmarks repo, which will give a side-by-side comparison once it is merged & re-run. For now, here are results from running everything on my machine: table.md

If i’m reading the results correctly, the final document is around 800KB compared to 250KB with automerge or others.

Yes, by default the saved states use a lazy JSON encoding, which is less efficient than the binary encodings used by other libraries. In principle,one could make a JSON-to-protobuf encoding, but for now, I've just been using GZIP.

You can also reduce the JSON size from 800->500KB by using the Text class instead of a List of single characters: https://github.com/mweidner037/list-positions/blob/master/benchmark_results.md#text-direct