Closed chenyan-dfinity closed 1 year ago
Note The flamegraph link only works after you merge. Unchanged benchmarks are omitted.
Measure different collection libraries written in both Motoko and Rust.
The library names with _rs
suffix are written in Rust; the rest are written in Motoko.
We use the same random number generator with fixed seed to ensure that all collections contain the same elements, and the queries are exactly the same. Below we explain the measurements of each column in the table:
rts_max_live_size
after generate
call; For Rust, it reports the Wasm's memory page * 32Kb.O(10000 nlogn)
algorithm hitting the limit, while an O(n^2)
algorithm runs just fine.Note
- The Candid interface of the benchmark is minimal, therefore the serialization cost is negligible in this measurement.
- Due to the instrumentation overhead and cycle limit, we cannot profile computations with large collections. Hopefully, when deterministic time slicing is ready, we can measure the performance on larger memory footprint.
hashmap
uses amortized data structure. When the initial capacity is reached, it has to copy the whole array, thus the cost ofbatch_put 50
is much higher than other data structures.hashmap_rs
uses thefxhash
crate, which is the same asstd::collections::HashMap
, but with a deterministic hasher. This ensures reproducible result.btree
comes from Byron Becker's stable BTreeMap library.zhenya_hashmap
comes from Zhenya Usenko's stable HashMap library.- The MoVM table measures the performance of an experimental implementation of Motoko interpreter. External developers can ignore this table for now.
binary_size | generate 50k | max mem | batch_get 50 | batch_put 50 | batch_remove 50 | |
---|---|---|---|---|---|---|
hashmap | 152_580 | 1_195_632_150 | 9_102_052 | 545_645 | 365_569_669 | 520_876 |
triemap | 156_424 | 1_338_995_779 | 9_715_900 | 459_710 | 1_193_026 | 686_569 |
rbtree | 153_258 | 1_115_533_975 | 8_902_160 | 354_721 | 964_237 | 495_133 |
splay | 152_693 | 1_323_550_652 | 8_702_096 | 719_103 | 1_214_198 | 717_146 |
btree | 180_227 | 1_222_588_229 | 7_556_172 | 502_876 | 1_090_262 | 540_393 |
zhenya_hashmap | 148_470 | 989_558_312 | 9_301_800 | 334_927 | 818_203 | 335_264 |
btreemap_rs | 526_004 | 129_727_131 | 1_638_400 | 76_248 | 154_643 | 75_646 |
hashmap_rs | 516_249 | 57_891_990 | 1_835_008 | 27_213 | 69_735 | 28_539 |
binary_size | heapify 50k | mem | pop_min 50 | put 50 | ||
---|---|---|---|---|---|---|
heap | 139_951 | 369_466_193 | 1_400_024 | 334_365 | 397_474 | 335_750 |
heap_rs | 498_010 | 7_208_921 | 819_200 | 52_887 | 27_273 | 53_085 |
binary_size | generate 10k | max mem | batch_get 50 | batch_put 50 | batch_remove 50 | |
---|---|---|---|---|---|---|
hashmap | 152_580 | 238_966_334 | 1_820_844 | 543_937 | 73_525_914 | 518_626 |
hashmap_rs | 516_249 | 11_916_012 | 950_272 | 26_546 | 68_942 | 27_406 |
imrc_hashmap_rs | 524_212 | 27_848_989 | 1_572_864 | 39_010 | 165_715 | 49_063 |
movm_rs | 1_584_149 | 1_255_894_005 | 2_654_208 | 3_010_534 | 8_080_808 | 6_407_407 |
movm_dynamic_rs | 1_597_032 | 573_543_536 | 2_129_920 | 2_325_226 | 3_157_234 | 2_308_078 |
Measure the performance of some typical dapps:
heartbeat
disabled to make profiling easier. We have a separate benchmark to measure heartbeat performance.Note
- The cost difference is mainly due to the Candid serialization cost.
- Motoko statically compiles/specializes the serialization code for each method, whereas in Rust, we use
serde
to dynamically deserialize data based on data on the wire.- We could improve the performance on the Rust side by using parser combinators. But it is a challenge to maintain the ergonomics provided by
serde
.- For real-world applications, we tend to send small data for each endpoint, which makes the Candid overhead in Rust tolerable.
binary_size | init | transfer_token | submit_proposal | vote_proposal | |
---|---|---|---|---|---|
Motoko | 225_805 | 37_493 | 16_270 | 12_654 | 14_126 |
Rust | 757_835 | 657_991 | 121_955 | 144_498 | 160_898 |
binary_size | init | mint_token | transfer_token | |
---|---|---|---|---|
Motoko | 183_882 | 12_181 | 22_319 | 4_710 |
Rust | 832_938 | 170_821 | 442_716 | 118_903 |
Measure the cost of empty heartbeat and timer job.
setTimer
measures both the setTimer(0)
method and the execution of empty job.setTimer
and canister_global_timer
function. If it's not there, we may need to adjust the script.binary_size | heartbeat | |
---|---|---|
Motoko | 118_909 | 7_392 |
Rust | 30_514 | 637 |
binary_size | setTimer | cancelTimer | |
---|---|---|---|
Motoko | 125_168 | 15_208 | 1_679 |
Rust | 525_902 | 70_091 | 14_399 |
Measure the cost of inter-canister calls from the Publisher & Subscriber example.
pub_binary_size | sub_binary_size | subscribe_caller | subscribe_callee | publish_caller | publish_callee | |
---|---|---|---|---|---|---|
Motoko | 139_886 | 126_827 | 14_632 | 8_451 | 10_530 | 3_662 |
Rust | 519_924 | 579_400 | 76_157 | 51_221 | 100_740 | 60_192 |
Map
Priority queue
MoVM
Basic DAO
DIP721 NFT
Heartbeat
Timer
Publisher & Subscriber