zxch3n
commented
9 months ago Hi @dmonad!
Thanks for diving into this. I've created a PR to fix the implementation #25
Latest Loro Benchmark Results
Ran on a 13-inch M1 MacBook Pro 2020 |N = 6000 | loro| | :- | -: | |Version | 0.10.1 | |Bundle size | 1,052,250 bytes | |Bundle size (gzipped) | 399,686 bytes | |[B1.1] Append N characters (time) | 218 ms | |[B1.1] Append N characters (avgUpdateSize) | 109 bytes | |[B1.1] Append N characters (encodeTime) | 0 ms | |[B1.1] Append N characters (docSize) | 6,162 bytes | |[B1.1] Append N characters (memUsed) | 353.1 kB | |[B1.1] Append N characters (parseTime) | 45 ms | |[B1.2] Insert string of length N (time) | 0 ms | |[B1.2] Insert string of length N (avgUpdateSize) | 6,107 bytes | |[B1.2] Insert string of length N (encodeTime) | 0 ms | |[B1.2] Insert string of length N (docSize) | 6,117 bytes | |[B1.2] Insert string of length N (memUsed) | 0 B | |[B1.2] Insert string of length N (parseTime) | 40 ms | |[B1.3] Prepend N characters (time) | 49 ms | |[B1.3] Prepend N characters (avgUpdateSize) | 108 bytes | |[B1.3] Prepend N characters (encodeTime) | 16 ms | |[B1.3] Prepend N characters (docSize) | 12,125 bytes | |[B1.3] Prepend N characters (memUsed) | 0 B | |[B1.3] Prepend N characters (parseTime) | 54 ms | |[B1.4] Insert N characters at random positions (time) | 39 ms | |[B1.4] Insert N characters at random positions (avgUpdateSize) | 109 bytes | |[B1.4] Insert N characters at random positions (encodeTime) | 23 ms | |[B1.4] Insert N characters at random positions (docSize) | 35,401 bytes | |[B1.4] Insert N characters at random positions (memUsed) | 0 B | |[B1.4] Insert N characters at random positions (parseTime) | 33 ms | |[B1.5] Insert N words at random positions (time) | 38 ms | |[B1.5] Insert N words at random positions (avgUpdateSize) | 117 bytes | |[B1.5] Insert N words at random positions (encodeTime) | 46 ms | |[B1.5] Insert N words at random positions (docSize) | 94,524 bytes | |[B1.5] Insert N words at random positions (memUsed) | 0 B | |[B1.5] Insert N words at random positions (parseTime) | 31 ms | |[B1.6] Insert string, then delete it (time) | 1 ms | |[B1.6] Insert string, then delete it (avgUpdateSize) | 6,217 bytes | |[B1.6] Insert string, then delete it (encodeTime) | 0 ms | |[B1.6] Insert string, then delete it (docSize) | 6,120 bytes | |[B1.6] Insert string, then delete it (memUsed) | 2.6 kB | |[B1.6] Insert string, then delete it (parseTime) | 29 ms | |[B1.7] Insert/Delete strings at random positions (time) | 46 ms | |[B1.7] Insert/Delete strings at random positions (avgUpdateSize) | 113 bytes | |[B1.7] Insert/Delete strings at random positions (encodeTime) | 11 ms | |[B1.7] Insert/Delete strings at random positions (docSize) | 50,836 bytes | |[B1.7] Insert/Delete strings at random positions (memUsed) | 0 B | |[B1.7] Insert/Delete strings at random positions (parseTime) | 27 ms | |[B1.8] Append N numbers (time) | 41 ms | |[B1.8] Append N numbers (avgUpdateSize) | 114 bytes | |[B1.8] Append N numbers (encodeTime) | 1 ms | |[B1.8] Append N numbers (docSize) | 35,719 bytes | |[B1.8] Append N numbers (memUsed) | 0 B | |[B1.8] Append N numbers (parseTime) | 33 ms | |[B1.9] Insert Array of N numbers (time) | 9 ms | |[B1.9] Insert Array of N numbers (avgUpdateSize) | 35,735 bytes | |[B1.9] Insert Array of N numbers (encodeTime) | 1 ms | |[B1.9] Insert Array of N numbers (docSize) | 35,742 bytes | |[B1.9] Insert Array of N numbers (memUsed) | 0 B | |[B1.9] Insert Array of N numbers (parseTime) | 28 ms | |[B1.10] Prepend N numbers (time) | 40 ms | |[B1.10] Prepend N numbers (avgUpdateSize) | 113 bytes | |[B1.10] Prepend N numbers (encodeTime) | 6 ms | |[B1.10] Prepend N numbers (docSize) | 41,748 bytes | |[B1.10] Prepend N numbers (memUsed) | 0 B | |[B1.10] Prepend N numbers (parseTime) | 28 ms | |[B1.11] Insert N numbers at random positions (time) | 39 ms | |[B1.11] Insert N numbers at random positions (avgUpdateSize) | 114 bytes | |[B1.11] Insert N numbers at random positions (encodeTime) | 24 ms | |[B1.11] Insert N numbers at random positions (docSize) | 65,016 bytes | |[B1.11] Insert N numbers at random positions (memUsed) | 0 B | |[B1.11] Insert N numbers at random positions (parseTime) | 28 ms | |[B2.1] Concurrently insert string of length N at index 0 (time) | 1 ms | |[B2.1] Concurrently insert string of length N at index 0 (updateSize) | 9,276 bytes | |[B2.1] Concurrently insert string of length N at index 0 (encodeTime) | 0 ms | |[B2.1] Concurrently insert string of length N at index 0 (docSize) | 12,248 bytes | |[B2.1] Concurrently insert string of length N at index 0 (memUsed) | 1.9 kB | |[B2.1] Concurrently insert string of length N at index 0 (parseTime) | 28 ms | |[B2.2] Concurrently insert N characters at random positions (time) | 204 ms | |[B2.2] Concurrently insert N characters at random positions (updateSize) | 665,563 bytes | |[B2.2] Concurrently insert N characters at random positions (encodeTime) | 57 ms | |[B2.2] Concurrently insert N characters at random positions (docSize) | 71,858 bytes | |[B2.2] Concurrently insert N characters at random positions (memUsed) | 0 B | |[B2.2] Concurrently insert N characters at random positions (parseTime) | 53 ms | |[B2.3] Concurrently insert N words at random positions (time) | 213 ms | |[B2.3] Concurrently insert N words at random positions (updateSize) | 731,420 bytes | |[B2.3] Concurrently insert N words at random positions (encodeTime) | 98 ms | |[B2.3] Concurrently insert N words at random positions (docSize) | 188,458 bytes | |[B2.3] Concurrently insert N words at random positions (memUsed) | 0 B | |[B2.3] Concurrently insert N words at random positions (parseTime) | 49 ms | |[B2.4] Concurrently insert & delete (time) | 427 ms | |[B2.4] Concurrently insert & delete (updateSize) | 1,441,427 bytes | |[B2.4] Concurrently insert & delete (encodeTime) | 155 ms | |[B2.4] Concurrently insert & delete (docSize) | 289,590 bytes | |[B2.4] Concurrently insert & delete (memUsed) | 0 B | |[B2.4] Concurrently insert & delete (parseTime) | 53 ms | |[B3.1] 20√N clients concurrently set number in Map (time) | 43 ms | |[B3.1] 20√N clients concurrently set number in Map (updateSize) | 161,636 bytes | |[B3.1] 20√N clients concurrently set number in Map (encodeTime) | 3 ms | |[B3.1] 20√N clients concurrently set number in Map (docSize) | 21,506 bytes | |[B3.1] 20√N clients concurrently set number in Map (memUsed) | 0 B | |[B3.1] 20√N clients concurrently set number in Map (parseTime) | 46 ms | |[B3.2] 20√N clients concurrently set Object in Map (time) | 50 ms | |[B3.2] 20√N clients concurrently set Object in Map (updateSize) | 200,630 bytes | |[B3.2] 20√N clients concurrently set Object in Map (encodeTime) | 3 ms | |[B3.2] 20√N clients concurrently set Object in Map (docSize) | 40,494 bytes | |[B3.2] 20√N clients concurrently set Object in Map (memUsed) | 584 B | |[B3.2] 20√N clients concurrently set Object in Map (parseTime) | 75 ms | |[B3.3] 20√N clients concurrently set String in Map (time) | 115 ms | |[B3.3] 20√N clients concurrently set String in Map (updateSize) | 7,940,240 bytes | |[B3.3] 20√N clients concurrently set String in Map (encodeTime) | 46 ms | |[B3.3] 20√N clients concurrently set String in Map (docSize) | 7,798,572 bytes | |[B3.3] 20√N clients concurrently set String in Map (memUsed) | 488 B | |[B3.3] 20√N clients concurrently set String in Map (parseTime) | 51 ms | |[B3.4] 20√N clients concurrently insert text in Array (time) | 154 ms | |[B3.4] 20√N clients concurrently insert text in Array (updateSize) | 166,750 bytes | |[B3.4] 20√N clients concurrently insert text in Array (encodeTime) | 9 ms | |[B3.4] 20√N clients concurrently insert text in Array (docSize) | 31,098 bytes | |[B3.4] 20√N clients concurrently insert text in Array (memUsed) | 0 B | |[B3.4] 20√N clients concurrently insert text in Array (parseTime) | 52 ms | |[B4] Apply real-world editing dataset (time) | 191 ms | |[B4] Apply real-world editing dataset (encodeTime) | 49 ms | |[B4] Apply real-world editing dataset (docSize) | 258,228 bytes | |[B4] Apply real-world editing dataset (parseTime) | 8 ms | |[B4] Apply real-world editing dataset (memUsed) | 0 B | |[B4x100] Apply real-world editing dataset 100 times (time) | 20,540 ms | |[B4x100] Apply real-world editing dataset 100 times (encodeTime) | 9,445 ms | |[B4x100] Apply real-world editing dataset 100 times (docSize) | 25,805,795 bytes | |[B4x100] Apply real-world editing dataset 100 times (parseTime) | 852 ms | |[B4x100] Apply real-world editing dataset 100 times (memUsed) | 2.8 kB |However, you can still see some discrepancies with the numbers reported on the website because the website reports benchmark data for the version dated 2023-11-08. Over the past two months, we've done a lot of refactoring and adjustments, the most significant of which is the change in our encoding format https://github.com/loro-dev/loro/pull/219. Additionally, we switched the compilation optimization level (opt-level) for WASM from 3 to 's', which may result in some performance trade-offs for a smaller WASM size. These refactorings will lead to changes in many performance metrics, and we have not yet embarked on more specific optimization work.
I think it's fair to use exportSnapshot during encoding because Loro's exportSnapshot actually serves a similar purpose to Automerge's save. It efficiently encodes both state and history, so in terms of document size, it's similar to exportUpdates. Our method is not exporting the internal memory representation; if it were, the document size would likely be much larger.
dmonad
Hi @loro-dev, @zxch3n @Leeeon233,
After looking at the benchmarks at your website, I wanted to reproduce your results. Unfortunately, I can't find your fork of my benchmarking suite in your organization. So I integrated Loro in my benchmarking suite. Please feel free to maintain your benchmarks in this repository!
My benchmark results look very different from what you stated on your website. While you state that
parseTimeandtimeare almost (a lot) smaller than Yjs, I find the exact opposite here. In some cases, Loro seems to have exponential runtime behavior (e.g. in the B2 & B3 benchmarks, bothtimeandparseTimeincrease exponentially with N). B4x100 takes several hours to execute, not just mere seconds as you stated.Can you please explain the disparity between these results? Is there some hidden API that I'm supposed to use?
Below, I compiled a table that shows the disparity.
Disparity between benchmark results
Loro Website Benchmarks (extracted January 23, 2024)