search

dmonad / crdt-benchmarks

A collection of CRDT benchmarks
Other
443 stars 29 forks source link

Add Operation Transformation approach #7

Open dmonad opened 4 years ago

dmonad commented 4 years ago

More or less the expected results. OT dominates when no transformations are involved. Yjs performs better when operations need to be transformed ([B2]).

TODO

Preliminary results

@josephg

N = 6000 Yjs Automerge delta-crdts OT
Version 13.3.0 0.14.1 0.10.3 4.0.0
Bundle size 65923 bytes 259763 bytes 227573 bytes 10540 bytes
Bundle size (gzipped) 19377 bytes 61478 bytes 64388 bytes 3805 bytes
[B1.1] Append N characters (time) 325 ms 27 ms
[B1.1] Append N characters (avgUpdateSize) 27 bytes 11 bytes
[B1.1] Append N characters (docSize) 6031 bytes 6004 bytes
[B1.1] Append N characters (memUsed) 1.6 MB 10.3 MB
[B1.1] Append N characters (parseTime) 1 ms 0 ms
[B1.1] Append N characters skipping skipping
[B1.2] Insert string of length N (time) 2 ms 0 ms
[B1.2] Insert string of length N (avgUpdateSize) 6031 bytes 6006 bytes
[B1.2] Insert string of length N (docSize) 6031 bytes 6004 bytes
[B1.2] Insert string of length N (memUsed) 271.8 kB 110 kB
[B1.2] Insert string of length N (parseTime) 0 ms 0 ms
[B1.2] Insert string of length N skipping skipping
[B1.3] Prepend N characters (time) 264 ms 48 ms
[B1.3] Prepend N characters (avgUpdateSize) 27 bytes 6 bytes
[B1.3] Prepend N characters (docSize) 6041 bytes 6004 bytes
[B1.3] Prepend N characters (memUsed) 2.5 MB 1.3 MB
[B1.3] Prepend N characters (parseTime) 27 ms 0 ms
[B1.3] Prepend N characters skipping skipping
[B1.4] Insert N characters at random positions (time) 257 ms 28 ms
[B1.4] Insert N characters at random positions (avgUpdateSize) 29 bytes 10 bytes
[B1.4] Insert N characters at random positions (docSize) 29614 bytes 6004 bytes
[B1.4] Insert N characters at random positions (memUsed) 0 B 60.3 kB
[B1.4] Insert N characters at random positions (parseTime) 2 ms 0 ms
[B1.4] Insert N characters at random positions skipping skipping
[B1.5] Insert N words at random positions (time) 234 ms 22 ms
[B1.5] Insert N words at random positions (avgUpdateSize) 36 bytes 16 bytes
[B1.5] Insert N words at random positions (docSize) 87826 bytes 36198 bytes
[B1.5] Insert N words at random positions (memUsed) 1.4 MB 1.1 MB
[B1.5] Insert N words at random positions (parseTime) 7 ms 0 ms
[B1.5] Insert N words at random positions skipping skipping
[B1.6] Insert string, then delete it (time) 1 ms 0 ms
[B1.6] Insert string, then delete it (avgUpdateSize) 6053 bytes 6019 bytes
[B1.6] Insert string, then delete it (docSize) 38 bytes 2 bytes
[B1.6] Insert string, then delete it (memUsed) 204.4 kB 60.4 kB
[B1.6] Insert string, then delete it (parseTime) 0 ms 0 ms
[B1.6] Insert string, then delete it skipping skipping
[B1.7] Insert/Delete strings at random positions (time) 272 ms 24 ms
[B1.7] Insert/Delete strings at random positions (avgUpdateSize) 31 bytes 14 bytes
[B1.7] Insert/Delete strings at random positions (docSize) 28691 bytes 2512 bytes
[B1.7] Insert/Delete strings at random positions (memUsed) 15.4 MB 0 B
[B1.7] Insert/Delete strings at random positions (parseTime) 6 ms 0 ms
[B1.7] Insert/Delete strings at random positions skipping skipping
[B1.8] Append N numbers (time) 268 ms
[B1.8] Append N numbers (avgUpdateSize) 32 bytes
[B1.8] Append N numbers (docSize) 35634 bytes
[B1.8] Append N numbers (memUsed) 0 B
[B1.8] Append N numbers (parseTime) 0 ms
[B1.8] Append N numbers skipping skipping
[B1.9] Insert Array of N numbers (time) 2 ms
[B1.9] Insert Array of N numbers (avgUpdateSize) 35661 bytes
[B1.9] Insert Array of N numbers (docSize) 35661 bytes
[B1.9] Insert Array of N numbers (memUsed) 3 MB
[B1.9] Insert Array of N numbers (parseTime) 0 ms
[B1.9] Insert Array of N numbers skipping skipping
[B1.10] Prepend N numbers (time) 159 ms
[B1.10] Prepend N numbers (avgUpdateSize) 32 bytes
[B1.10] Prepend N numbers (docSize) 35669 bytes
[B1.10] Prepend N numbers (memUsed) 5.1 MB
[B1.10] Prepend N numbers (parseTime) 2 ms
[B1.10] Prepend N numbers skipping skipping
[B1.11] Insert N numbers at random positions (time) 172 ms
[B1.11] Insert N numbers at random positions (avgUpdateSize) 34 bytes
[B1.11] Insert N numbers at random positions (docSize) 59161 bytes
[B1.11] Insert N numbers at random positions (memUsed) 6.1 MB
[B1.11] Insert N numbers at random positions (parseTime) 2 ms
[B1.11] Insert N numbers at random positions skipping skipping
[B2.1] Concurrently insert string of length N at index 0 (time) 1 ms 1 ms
[B2.1] Concurrently insert string of length N at index 0 (updateSize) 12058 bytes 12012 bytes
[B2.1] Concurrently insert string of length N at index 0 (docSize) 12149 bytes 12104 bytes
[B2.1] Concurrently insert string of length N at index 0 (memUsed) 523.6 kB 171.5 kB
[B2.1] Concurrently insert string of length N at index 0 (parseTime) 0 ms 0 ms
[B2.1] Concurrently insert string of length N at index 0 skipping skipping
[B2.2] Concurrently insert N characters at random positions (time) 76 ms 14190 ms
[B2.2] Concurrently insert N characters at random positions (updateSize) 66360 bytes 125734 bytes
[B2.2] Concurrently insert N characters at random positions (docSize) 66454 bytes 12104 bytes
[B2.2] Concurrently insert N characters at random positions (memUsed) 10 MB 13.7 MB
[B2.2] Concurrently insert N characters at random positions (parseTime) 6 ms 0 ms
[B2.2] Concurrently insert N characters at random positions skipping skipping
[B2.3] Concurrently insert N words at random positions (time) 80 ms 16041 ms
[B2.3] Concurrently insert N words at random positions (updateSize) 177753 bytes 194342 bytes
[B2.3] Concurrently insert N words at random positions (docSize) 177918 bytes 71752 bytes
[B2.3] Concurrently insert N words at random positions (memUsed) 7.7 MB 10.7 MB
[B2.3] Concurrently insert N words at random positions (parseTime) 30 ms 0 ms
[B2.3] Concurrently insert N words at random positions skipping skipping
[B2.4] Concurrently insert & delete (time) 156 ms 68234 ms
[B2.4] Concurrently insert & delete (updateSize) 278025 bytes 383898 bytes
[B2.4] Concurrently insert & delete (docSize) 278153 bytes 77765 bytes
[B2.4] Concurrently insert & delete (memUsed) 16.5 MB 14.4 MB
[B2.4] Concurrently insert & delete (parseTime) 33 ms 0 ms
[B2.4] Concurrently insert & delete skipping skipping
[B3.1] 20√N clients concurrently set number in Map (time) 490 ms
[B3.1] 20√N clients concurrently set number in Map (updateSize) 49174 bytes
[B3.1] 20√N clients concurrently set number in Map (docSize) 32232 bytes
[B3.1] 20√N clients concurrently set number in Map (memUsed) 17.1 MB
[B3.1] 20√N clients concurrently set number in Map (parseTime) 25 ms
[B3.1] 20√N clients concurrently set number in Map skipping
[B3.2] 20√N clients concurrently set Object in Map (time) 526 ms
[B3.2] 20√N clients concurrently set Object in Map (updateSize) 95862 bytes
[B3.2] 20√N clients concurrently set Object in Map (docSize) 41484 bytes
[B3.2] 20√N clients concurrently set Object in Map (memUsed) 6.7 MB
[B3.2] 20√N clients concurrently set Object in Map (parseTime) 21 ms
[B3.2] 20√N clients concurrently set Object in Map skipping
[B3.3] 20√N clients concurrently set String in Map (time) 618 ms
[B3.3] 20√N clients concurrently set String in Map (updateSize) 7826229 bytes
[B3.3] 20√N clients concurrently set String in Map (docSize) 36824 bytes
[B3.3] 20√N clients concurrently set String in Map (memUsed) 0 B
[B3.3] 20√N clients concurrently set String in Map (parseTime) 14 ms
[B3.3] 20√N clients concurrently set String in Map skipping
[B3.4] 20√N clients concurrently insert text in Array (time) 534 ms
[B3.4] 20√N clients concurrently insert text in Array (updateSize) 52741 bytes
[B3.4] 20√N clients concurrently insert text in Array (docSize) 26586 bytes
[B3.4] 20√N clients concurrently insert text in Array (memUsed) 14.4 MB
[B3.4] 20√N clients concurrently insert text in Array (parseTime) 4 ms
[B3.4] 20√N clients concurrently insert text in Array skipping skipping
[B4] Apply real-world editing dataset (time) 6006 ms 129 ms
[B4] Apply real-world editing dataset (avgUpdateSize) 29 bytes 13 bytes
[B4] Apply real-world editing dataset (encodeTime) 6 ms 1 ms
[B4] Apply real-world editing dataset (docSize) 159929 bytes 108923 bytes
[B4] Apply real-world editing dataset (memUsed) 7 MB 262.8 kB
[B4] Apply real-world editing dataset (parseTime) 9 ms 0 ms
[B4] Apply real-world editing dataset skipping skipping
[B4 x 100] Apply real-world editing dataset 100 times (time) 139701 ms 31756 ms
[B4 x 100] Apply real-world editing dataset 100 times (encodeTime) 616 ms 1092 ms
[B4 x 100] Apply real-world editing dataset 100 times (docSize) 15989245 bytes 10891904 bytes
[B4 x 100] Apply real-world editing dataset 100 times (parseTime) 1825 ms 1071 ms
[B4 x 100] Apply real-world editing dataset 100 times (memUsed) 466.1 MB 0 B
josephg commented 4 years ago

Oooh fascinating! <3