Gas costs post-memtrie

[Ekleog-NEAR]

With memtrie, we made our storage reads much faster. It already enabled stateless validation.

Now we want to lower at least compute costs, in order to also get single-shard performance improvements from that work.

In order to do that, we must retake ownership of parameters estimator, run it with the newest changes, and then consider what costs changes we should do.

[Ekleog-NEAR]

Notes from today's meeting with @tayfunelmas and @akhi3030:

• What are the costs we expect should change with memtrie?
  • Storage reads:
  • Read base/key byte/value byte, has key base/byte, TTN
  • Storage writes: it will first go through memtrie then to disk
  • Write *, Remove *
  • Don’t reevaluate the storage write costs for now, it’ll wait for when writes get out of the critical path, or even get eliminated completely altogether
• What are their expected new order of magnitude, eg. compared to memory reads?
  • Between 1.3x and 10x faster probably
• Estimator uses directly ShardTries, so it could be by default, but how can I confirm with certainty that I’m not measuring without memtrie enabled?
  • Action: send link to code to tayfun
• How can I make sure memtrie is enabled in the parameter estimator eg. in a profiler results?
  • There should be a function that we can check for in the profiler
  • Just look for this function: https://github.com/near/nearcore/blob/3d4f554288d196d37e91d1460323cc6e02c735c8/core/store/src/trie/mod.rs#L1366

---

• We can ignore qemu-based computation
• Run with and without shard cache
• What hardware to run on? N2-standard-8?
  • Yes for MVP, then consider running on multiple machines later maybe

[Ekleog-NEAR]

I ended up not being able to run on n2-standard-8 due to some illegal instructions, most likely the newer sha256 instructions due to n2 being too old a cpu architecture. Switched to the amd-based n2d-standard-8, and got a few runs out.

Command used, always with 1 warmup iter but with a number of actual iterations that varies:

time cargo run --release --package runtime-params-estimator --features required --bin runtime-params-estimator -- --accounts-num 20000 --additional-accounts-num 200000 --iters 1 --warmup-iters 1 --metric time --costs StorageWriteBase,StorageWriteKeyByte,StorageWriteValueByte,StorageWriteEvictedByte,StorageReadBase,StorageReadKeyByte,StorageReadValueByte,StorageRemoveBase,StorageRemoveKeyByte,StorageRemoveRetValueByte,StorageHasKeyBase,StorageHasKeyByte,TouchingTrieNode,ReadCachedTrieNode

1 iter

``` StorageHasKeyBase 12_900_400_629 gas [ 12.9µs] UNCERTAIN (computed in 140.18s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageHasKeyByte 32_974_229 gas [ 33ns] UNCERTAIN (computed in 48.62s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadBase 13_176_595_629 gas [ 13.177µs] UNCERTAIN (computed in 37.94s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadKeyByte 32_950_083 gas [ 33ns] UNCERTAIN (computed in 48.77s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadValueByte 1_996_159 gas [ 2ns] UNCERTAIN (computed in 42.34s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteBase 5_894_729_584 gas [ 5.895µs] UNCERTAIN (computed in 38.77s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteKeyByte 6_054_362 gas [ 6ns] UNCERTAIN (computed in 66.78s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteValueByte 2_372_167 gas [ 2ns] UNCERTAIN (computed in 40.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteEvictedByte 3_502_758 gas [ 4ns] UNCERTAIN (computed in 42.64s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveBase 13_325_130_629 gas [ 13.325µs] UNCERTAIN (computed in 38.65s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveKeyByte 12_766_994 gas [ 13ns] UNCERTAIN (computed in 50.31s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveRetValueByte 1_660_624 gas [ 2ns] UNCERTAIN (computed in 42.21s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 TouchingTrieNode 8_087_650_976 gas [ 8.088µs] UNCERTAIN (computed in 39.79s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 838_062_500 gas [ 838ns] UNCERTAIN (computed in 120.42s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 879.35s, output saved to: ```

2 iters

``` StorageHasKeyBase 15_193_494_159 gas [ 15.193µs] UNCERTAIN (computed in 116.28s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageHasKeyByte 35_605_815 gas [ 36ns] UNCERTAIN (computed in 50.08s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadBase 13_538_829_159 gas [ 13.539µs] UNCERTAIN (computed in 35.86s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadKeyByte 33_160_425 gas [ 33ns] UNCERTAIN (computed in 49.22s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadValueByte 2_006_289 gas [ 2ns] UNCERTAIN (computed in 41.24s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteBase 6_010_015_762 gas [ 6.01µs] UNCERTAIN (computed in 34.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteKeyByte 6_186_676 gas [ 6ns] UNCERTAIN (computed in 91.99s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteValueByte 2_401_317 gas [ 2ns] UNCERTAIN (computed in 38.72s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteEvictedByte 3_662_605 gas [ 4ns] UNCERTAIN (computed in 41.88s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveBase 13_380_131_159 gas [ 13.38µs] UNCERTAIN (computed in 35.62s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveKeyByte 12_818_790 gas [ 13ns] UNCERTAIN (computed in 52.65s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveRetValueByte 1_669_679 gas [ 2ns] UNCERTAIN (computed in 40.42s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 TouchingTrieNode 6_168_517_526 gas [ 6.169µs] UNCERTAIN (computed in 35.43s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 938_437_500 gas [ 938ns] (computed in 158.17s) Finished in 867.08s, output saved to: ```

3 iters

``` StorageHasKeyBase 13_065_329_313 gas [ 13.065µs] (computed in 109.02s) StorageHasKeyByte 32_983_184 gas [ 33ns] (computed in 55.80s) StorageReadBase 13_752_679_313 gas [ 13.753µs] (computed in 36.14s) StorageReadKeyByte 33_064_852 gas [ 33ns] (computed in 55.15s) StorageReadValueByte 2_080_969 gas [ 2ns] (computed in 45.52s) StorageWriteBase 5_988_201_695 gas [ 5.988µs] (computed in 36.06s) StorageWriteKeyByte 6_486_408 gas [ 6ns] (computed in 119.46s) StorageWriteValueByte 2_391_904 gas [ 2ns] (computed in 39.74s) StorageWriteEvictedByte 3_559_623 gas [ 4ns] (computed in 45.74s) StorageRemoveBase 13_027_891_147 gas [ 13.028µs] (computed in 35.79s) StorageRemoveKeyByte 12_733_717 gas [ 13ns] (computed in 60.51s) StorageRemoveRetValueByte 1_692_055 gas [ 2ns] (computed in 43.00s) TouchingTrieNode 8_995_546_152 gas [ 8.996µs] UNCERTAIN (computed in 35.47s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 897_291_666 gas [ 897ns] UNCERTAIN (computed in 198.09s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 960.90s, output saved to: ```

4 iters

``` StorageHasKeyBase 14_058_590_550 gas [ 14.059µs] (computed in 79.23s) StorageHasKeyByte 33_011_596 gas [ 33ns] (computed in 61.74s) StorageReadBase 14_448_919_175 gas [ 14.449µs] UNCERTAIN (computed in 36.10s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadKeyByte 32_979_505 gas [ 33ns] (computed in 60.98s) StorageReadValueByte 2_017_662 gas [ 2ns] (computed in 48.73s) StorageWriteBase 6_040_652_050 gas [ 6.041µs] (computed in 36.30s) StorageWriteKeyByte 6_248_627 gas [ 6ns] (computed in 148.50s) StorageWriteValueByte 2_347_736 gas [ 2ns] (computed in 42.35s) StorageWriteEvictedByte 3_590_032 gas [ 4ns] (computed in 48.67s) StorageRemoveBase 13_096_787_050 gas [ 13.097µs] (computed in 35.98s) StorageRemoveKeyByte 12_601_924 gas [ 13ns] (computed in 68.06s) StorageRemoveRetValueByte 1_640_915 gas [ 2ns] (computed in 45.55s) TouchingTrieNode 6_211_604_656 gas [ 6.212µs] (computed in 36.96s) ReadCachedTrieNode 971_890_625 gas [ 972ns] UNCERTAIN (computed in 240.09s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 1034.30s, output saved to: ```

5 iters

``` StorageHasKeyBase 13_483_639_212 gas [ 13.484µs] UNCERTAIN (computed in 75.35s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageHasKeyByte 32_955_855 gas [ 33ns] UNCERTAIN (computed in 66.64s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadBase 14_311_978_112 gas [ 14.312µs] UNCERTAIN (computed in 36.33s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadKeyByte 33_267_940 gas [ 33ns] UNCERTAIN (computed in 66.08s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadValueByte 2_027_850 gas [ 2ns] UNCERTAIN (computed in 10.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteBase 6_025_332_562 gas [ 6.025µs] UNCERTAIN (computed in 9.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteKeyByte 6_184_661 gas [ 6ns] UNCERTAIN (computed in 35.09s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteValueByte 2_391_077 gas [ 2ns] UNCERTAIN (computed in 11.00s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteEvictedByte 3_774_544 gas [ 4ns] UNCERTAIN (computed in 10.63s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveBase 14_726_878_112 gas [ 14.727µs] UNCERTAIN (computed in 9.43s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveKeyByte 12_753_175 gas [ 13ns] UNCERTAIN (computed in 13.90s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveRetValueByte 1_704_211 gas [ 2ns] UNCERTAIN (computed in 10.34s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 TouchingTrieNode 6_588_516_675 gas [ 6.589µs] (computed in 10.42s) ReadCachedTrieNode 1_069_487_500 gas [ 1.069µs] UNCERTAIN (computed in 254.29s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 664.60s, output saved to: ```

10 iters

``` StorageHasKeyBase 13_578_744_617 gas [ 13.579µs] UNCERTAIN (computed in 26.07s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageHasKeyByte 32_979_644 gas [ 33ns] UNCERTAIN (computed in 23.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadBase 13_407_996_617 gas [ 13.408µs] UNCERTAIN (computed in 8.48s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadKeyByte 33_113_673 gas [ 33ns] UNCERTAIN (computed in 20.91s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadValueByte 2_069_821 gas [ 2ns] UNCERTAIN (computed in 12.42s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteBase 6_324_529_522 gas [ 6.325µs] UNCERTAIN (computed in 9.55s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteKeyByte 6_161_299 gas [ 6ns] UNCERTAIN (computed in 61.94s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteValueByte 2_388_617 gas [ 2ns] UNCERTAIN (computed in 14.21s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteEvictedByte 3_691_684 gas [ 4ns] UNCERTAIN (computed in 12.58s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveBase 13_050_943_167 gas [ 13.051µs] UNCERTAIN (computed in 8.59s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveKeyByte 12_708_137 gas [ 13ns] UNCERTAIN (computed in 19.37s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveRetValueByte 1_690_279 gas [ 2ns] UNCERTAIN (computed in 11.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 TouchingTrieNode 7_055_293_690 gas [ 7.055µs] (computed in 10.44s) ReadCachedTrieNode 1_080_375_000 gas [ 1.08µs] UNCERTAIN (computed in 462.98s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 747.11s, output saved to: ```

20 iters

``` StorageHasKeyBase 13_534_877_379 gas [ 13.535µs] UNCERTAIN (computed in 26.32s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageHasKeyByte 32_954_422 gas [ 33ns] UNCERTAIN (computed in 36.62s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadBase 13_596_178_804 gas [ 13.596µs] UNCERTAIN (computed in 9.27s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadKeyByte 33_125_687 gas [ 33ns] UNCERTAIN (computed in 34.16s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadValueByte 2_107_915 gas [ 2ns] UNCERTAIN (computed in 16.41s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteBase 6_532_413_647 gas [ 6.532µs] UNCERTAIN (computed in 11.38s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteKeyByte 6_229_705 gas [ 6ns] UNCERTAIN (computed in 320.09s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteValueByte 2_400_606 gas [ 2ns] UNCERTAIN (computed in 86.49s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteEvictedByte 3_641_082 gas [ 4ns] UNCERTAIN (computed in 101.73s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveBase 13_505_353_779 gas [ 13.505µs] UNCERTAIN (computed in 41.04s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveKeyByte 12_778_403 gas [ 13ns] UNCERTAIN (computed in 192.44s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveRetValueByte 1_705_733 gas [ 2ns] UNCERTAIN (computed in 86.82s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 TouchingTrieNode 7_165_741_437 gas [ 7.166µs] UNCERTAIN (computed in 38.87s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 990_625_000 gas [ 991ns] UNCERTAIN (computed in 906.51s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 1952.98s, output saved to: ```

30 iters

``` StorageHasKeyBase 13_672_466_157 gas [ 13.672µs] UNCERTAIN (computed in 28.76s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageHasKeyByte 32_923_901 gas [ 33ns] UNCERTAIN (computed in 185.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadBase 13_992_991_990 gas [ 13.993µs] UNCERTAIN (computed in 46.02s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadKeyByte 32_970_357 gas [ 33ns] UNCERTAIN (computed in 214.10s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadValueByte 2_122_544 gas [ 2ns] UNCERTAIN (computed in 134.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteBase 6_609_279_470 gas [ 6.609µs] UNCERTAIN (computed in 54.12s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteKeyByte 6_356_957 gas [ 6ns] UNCERTAIN (computed in 923.27s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteValueByte 2_400_035 gas [ 2ns] UNCERTAIN (computed in 117.73s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteEvictedByte 3_977_237 gas [ 4ns] UNCERTAIN (computed in 135.38s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveBase 14_193_024_007 gas [ 14.193µs] UNCERTAIN (computed in 47.10s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveKeyByte 12_636_652 gas [ 13ns] UNCERTAIN (computed in 268.56s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveRetValueByte 1_692_422 gas [ 2ns] UNCERTAIN (computed in 112.93s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 TouchingTrieNode 7_553_665_014 gas [ 7.554µs] UNCERTAIN (computed in 43.51s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 929_081_250 gas [ 929ns] UNCERTAIN (computed in 1323.45s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 3679.74s, output saved to: ```

40 iters

``` StorageHasKeyBase 13_589_294_563 gas [ 13.589µs] (computed in 65.51s) StorageHasKeyByte 34_641_766 gas [ 35ns] UNCERTAIN (computed in 481.06s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadBase 14_583_977_776 gas [ 14.584µs] (computed in 47.28s) StorageReadKeyByte 32_810_104 gas [ 33ns] (computed in 274.82s) StorageReadValueByte 2_199_171 gas [ 2ns] (computed in 168.16s) StorageWriteBase 6_687_153_364 gas [ 6.687µs] (computed in 59.30s) StorageWriteKeyByte 6_298_567 gas [ 6ns] (computed in 1233.53s) StorageWriteValueByte 2_415_817 gas [ 2ns] (computed in 150.57s) StorageWriteEvictedByte 3_950_634 gas [ 4ns] UNCERTAIN (computed in 169.31s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveBase 13_956_983_851 gas [ 13.957µs] (computed in 51.10s) StorageRemoveKeyByte 12_773_246 gas [ 13ns] (computed in 345.62s) StorageRemoveRetValueByte 1_709_642 gas [ 2ns] (computed in 138.50s) TouchingTrieNode 13_769_501_151 gas [ 13.77µs] UNCERTAIN (computed in 45.87s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 980_657_812 gas [ 981ns] UNCERTAIN (computed in 1764.48s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 5040.11s, output saved to: ```

50 iters

``` StorageHasKeyBase 13_776_535_151 gas [ 13.777µs] UNCERTAIN (computed in 129.48s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageHasKeyByte 32_789_001 gas [ 33ns] (computed in 335.74s) StorageReadBase 14_081_979_751 gas [ 14.082µs] (computed in 51.99s) StorageReadKeyByte 33_072_709 gas [ 33ns] (computed in 335.58s) StorageReadValueByte 2_285_782 gas [ 2ns] UNCERTAIN (computed in 200.27s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteBase 6_764_527_654 gas [ 6.765µs] (computed in 67.03s) StorageWriteKeyByte 6_185_901 gas [ 6ns] (computed in 1551.28s) StorageWriteValueByte 2_697_848 gas [ 3ns] (computed in 186.35s) StorageWriteEvictedByte 5_259_693 gas [ 5ns] UNCERTAIN (computed in 201.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveBase 16_254_930_461 gas [ 16.255µs] UNCERTAIN (computed in 55.22s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveKeyByte 14_017_706 gas [ 14ns] (computed in 422.84s) StorageRemoveRetValueByte 1_818_455 gas [ 2ns] (computed in 164.58s) TouchingTrieNode 7_465_078_177 gas [ 7.465µs] UNCERTAIN (computed in 48.77s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 1_019_361_250 gas [ 1.019µs] UNCERTAIN (computed in 2203.93s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 5999.32s, output saved to: ```

75 iters

``` StorageHasKeyBase 14_014_411_258 gas [ 14.014µs] (computed in 198.15s) StorageHasKeyByte 32_971_225 gas [ 33ns] (computed in 494.61s) StorageReadBase 14_365_610_898 gas [ 14.366µs] (computed in 67.05s) StorageReadKeyByte 33_022_814 gas [ 33ns] (computed in 496.56s) StorageReadValueByte 2_350_673 gas [ 2ns] UNCERTAIN (computed in 373.15s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteBase 7_296_011_225 gas [ 7.296µs] UNCERTAIN (computed in 124.21s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteKeyByte 6_315_680 gas [ 6ns] (computed in 2434.43s) StorageWriteValueByte 2_538_901 gas [ 3ns] (computed in 292.03s) StorageWriteEvictedByte 4_342_591 gas [ 4ns] (computed in 295.30s) StorageRemoveBase 13_523_278_652 gas [ 13.523µs] (computed in 70.04s) StorageRemoveKeyByte 12_850_151 gas [ 13ns] (computed in 623.07s) StorageRemoveRetValueByte 1_765_050 gas [ 2ns] (computed in 234.38s) TouchingTrieNode 7_156_997_956 gas [ 7.157µs] UNCERTAIN (computed in 61.62s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 981_690_833 gas [ 982ns] UNCERTAIN (computed in 3445.15s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 9295.41s, output saved to: ```

100 iters

``` StorageHasKeyBase 19_351_911_321 gas [ 19.352µs] UNCERTAIN (computed in 220.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageHasKeyByte 33_052_763 gas [ 33ns] (computed in 657.38s) StorageReadBase 17_326_255_431 gas [ 17.326µs] UNCERTAIN (computed in 77.65s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadKeyByte 33_028_469 gas [ 33ns] (computed in 651.71s) StorageReadValueByte 2_226_692 gas [ 2ns] (computed in 382.95s) StorageWriteBase 7_246_144_383 gas [ 7.246µs] UNCERTAIN (computed in 108.05s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteKeyByte 6_034_886 gas [ 6ns] UNCERTAIN (computed in 3135.38s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteValueByte 2_500_958 gas [ 3ns] (computed in 400.69s) StorageWriteEvictedByte 4_125_927 gas [ 4ns] (computed in 386.37s) StorageRemoveBase 16_513_323_191 gas [ 16.513µs] UNCERTAIN (computed in 83.15s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveKeyByte 12_870_466 gas [ 13ns] (computed in 817.01s) StorageRemoveRetValueByte 2_041_085 gas [ 2ns] UNCERTAIN (computed in 299.20s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 TouchingTrieNode 8_713_369_479 gas [ 8.713µs] UNCERTAIN (computed in 69.04s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 960_275_000 gas [ 960ns] (computed in 4728.84s) Finished in 12105.58s, output saved to: ```

150 iters

``` StorageHasKeyBase 14_480_452_840 gas [ 14.48µs] UNCERTAIN (computed in 284.72s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageHasKeyByte 33_187_697 gas [ 33ns] (computed in 956.61s) StorageReadBase 18_059_659_117 gas [ 18.06µs] UNCERTAIN (computed in 104.38s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageReadKeyByte 33_280_597 gas [ 33ns] (computed in 986.92s) StorageReadValueByte 2_325_098 gas [ 2ns] (computed in 576.71s) StorageWriteBase 8_026_454_400 gas [ 8.026µs] UNCERTAIN (computed in 153.31s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteKeyByte 6_700_411 gas [ 7ns] UNCERTAIN (computed in 5587.91s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageWriteValueByte 2_529_798 gas [ 3ns] (computed in 639.25s) StorageWriteEvictedByte 4_247_221 gas [ 4ns] (computed in 582.29s) StorageRemoveBase 19_840_170_697 gas [ 19.84µs] UNCERTAIN (computed in 111.16s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 StorageRemoveKeyByte 13_153_151 gas [ 13ns] (computed in 1202.99s) StorageRemoveRetValueByte 1_979_379 gas [ 2ns] (computed in 430.68s) TouchingTrieNode 7_543_821_926 gas [ 7.544µs] UNCERTAIN (computed in 86.86s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24 ReadCachedTrieNode 1_095_268_333 gas [ 1.095µs] UNCERTAIN (computed in 7482.10s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13 Finished in 19273.83s, output saved to: ```

[Ekleog-NEAR]

Data out of the runs (without memtrie enabled yet)

Skill unlocked: using chatgpt to turn unstructured data into structured data:

Iteration	StorageHasKeyBase (Gas)	StorageHasKeyByte (Gas)	StorageReadBase (Gas)	StorageReadKeyByte (Gas)	StorageReadValueByte (Gas)	StorageWriteBase (Gas)	StorageWriteKeyByte (Gas)	StorageWriteValueByte (Gas)	StorageWriteEvictedByte (Gas)	StorageRemoveBase (Gas)	StorageRemoveKeyByte (Gas)	StorageRemoveRetValueByte (Gas)	TouchingTrieNode (Gas)	ReadCachedTrieNode (Gas)	Computation Time (Seconds)
1	12,900,400,629	32,974,229	13,176,595,629	32,950,083	1,996,159	5,894,729,584	6,054,362	2,372,167	3,502,758	13,325,130,629	12,766,994	1,660,624	8,087,650,976	838,062,500	879.35
2	15,193,494,159	35,605,815	13,538,829,159	33,160,425	2,006,289	6,010,015,762	6,186,676	2,401,317	3,662,605	13,380,131,159	12,818,790	1,669,679	6,168,517,526	938,437,500	867.08
3	13,065,329,313	32,983,184	13,752,679,313	33,064,852	2,080,969	5,988,201,695	6,486,408	2,391,904	3,559,623	13,027,891,147	12,733,717	1,692,055	8,995,546,152	897,291,666	960.9
4	14,058,590,550	33,011,596	14,448,919,175	32,979,505	2,017,662	6,040,652,050	6,248,627	2,347,736	3,590,032	13,096,787,050	12,601,924	1,640,915	6,211,604,656	971,890,625	1034.3
5	13,483,639,212	32,955,855	14,311,978,112	33,267,940	2,027,850	6,025,332,562	6,184,661	2,391,077	3,774,544	14,726,878,112	12,753,175	1,704,211	6,588,516,675	1,069,487,500	664.6
10	13,578,744,617	32,979,644	13,407,996,617	33,113,673	2,069,821	6,324,529,522	6,161,299	2,388,617	3,691,684	13,050,943,167	12,708,137	1,690,279	7,055,293,690	1,080,375,000	747.11
20	13,534,877,379	32,954,422	13,596,178,804	33,125,687	2,107,915	6,532,413,647	6,229,705	2,400,606	3,641,082	13,505,353,779	12,778,403	1,705,733	7,165,741,437	990,625,000	1952.98
30	13,672,466,157	32,923,901	13,992,991,990	32,970,357	2,122,544	6,609,279,470	6,356,957	2,400,035	3,977,237	14,193,024,007	12,636,652	1,692,422	7,553,665,014	929,081,250	3679.74
40	13,589,294,563	34,641,766	14,583,977,776	32,810,104	2,199,171	6,687,153,364	6,298,567	2,415,817	3,950,634	13,956,983,851	12,773,246	1,709,642	13,769,501,151	980,657,812	5040.11
50	13,776,535,151	32,789,001	14,081,979,751	33,072,709	2,285,782	6,764,527,654	6,185,901	2,697,848	5,259,693	16,254,930,461	14,017,706	1,818,455	7,465,078,177	1,019,361,250	5999.32
75	14,014,411,258	32,971,225	14,365,610,898	33,022,814	2,350,673	7,296,011,225	6,315,680	2,538,901	4,342,591	13,523,278,652	12,850,151	1,765,050	7,156,997,956	981,690,833	9295.41
100	19,351,911,321	33,052,763	17,326,255,431	33,028,469	2,226,692	7,246,144,383	6,034,886	2,500,958	4,125,927	16,513,323,191	12,870,466	2,041,085	8,713,369,479	960,275,000	12105.58
150	14,480,452,840	33,187,697	18,059,659,117	33,280,597	2,325,098	8,026,454,400	6,700,411	2,529,798	4,247,221	19,840,170,697	13,153,151	1,979,379	7,543,821,926	1,095,268,333	19273.83
Current Costs	200,000,000,000	30,790,845	200,000,000,000	30,952,533	5,611,005	200,000,000,000	70,482,867	31,018,539	32,117,307	200,000,000,000	38,220,384	11,531,556	110,000,000,000	2,280,000,000

This gives the following graphs:

---

In particular, it shows that while the results are mostly stable over a different number of iterations, the remaining instability seems to not be particularly lower when running more iterations. So the sweet spot will likely be running like ten 10-iters runs and doing data magic on them to decide what's good for us.

There is a slight ramp-up when increasing the iteration number, probably due to variance being better-detected and thus pushing the numbers up. This is likely not very relevant, as we should take a safety margin of more than this number.

---

So currently, our costs are set quite a bit higher than estimator results. If I understand Jakob's analysis correctly, this was done in order to derisk the flat storage launch: we wanted to be sure that our costs were not underestimated in the worst case.

AFAIU we're changing that policy nowadays to say that costs that just avoid missed chunks with our real-world load should be enough. Still, considering we're looking into various ways of getting state writes out of the critical path altogether, I'd suggest waiting until that happens to pursue gas costs reductions in writes: once flushing to disk is no longer a risk, we can safely ignore all the computations Jakob did in #8006 and just set values to the estimator-provided values.

---

Next steps

This project started around reevaluating the read costs with memtrie, so we should focus on that. But then we should likely continue with reducing the write costs too. My planned next steps are:

1. Update the estimator to enable memtrie
2. Confirm that the speedup order of magnitude is as expected
3. Pick some prospective gas costs for reads (this won't need as much care as flat storage, because memtrie performance should be much more stable and much less dependent on filesystem caches etc.)
4. Implement #11944
5. Use it to validate the prospective gas costs
6. If confirmed, perform the full protocol change and deploy the new gas costs in the next release

[Ekleog-NEAR]

Here is what the numbers of the parameter estimator out of 12adab4 (with the log line removed) look like:

(The full data for the runs used for these graphs is available here)

So memtrie does bring in very significant benefits on estimator results! Though it also seems to make writing a bit slower at times. Thanksfully we already have a very large safety margin on writes, so we should be able to just ignore the increase in costs here. And the most significant benefit of memtrie is that it removes all disk access from the code path, meaning that the variance should be much lower, and thus that we can take a much smaller safety margin compared to the previous 15G -> 200G safety margin that we needed to account for disk limits under load.

With all that in mind, I suggest targeting a roughly 20~30% safety margin above estimator 90th percentile, because memtrie should have a pretty low variance:

• StorageHasKeyBase: 200,000,000,000 -> 8,000,000,000, without 4k split because checking whether the key exists should AFAIU never require falling back to disk (⚠️ assumption that needs to be confirmed by someone familiar with memtrie)
• StorageHasKeyByte: 30,790,845 -> 9,000,000, without 4k split (⚠️ same assumption)
• StorageReadBase: 200,000,000,000 -> same if the value is 4k or more; 9,000,000,000 if the value is strictly less than 4k
• StorageReadKeyByte: 30,952,533 -> same if the value is 4k or more; 10,000,000 otherwise
• StorageReadValueByte: 5,611,005 -> same if the value is 4k or more; 2,500,000 otherwise
• StorageWrite*: do not change
• StorageRemove*: do not change
• TouchingTrieNode: 110,000,000,000 -> 6,000,000,000, without 4k split (⚠️ same assumption)
• ReadCachedTrieNode: 2,280,000,000 -> 1,400,000,000, we probably do not need the 2x safety margin that we originally had, even though memtrie did not make things much faster the estimator is still around 1,070,000,000 only

My next step will be making the protocol version change that does this split. It'll be my first time adding a cost so it might take a bit longer than initially expected.

@Longarithm / @tayfunelmas can you check in particular the assumptions in three places that have the ⚠️ sign? I'll also double-check them myself while making the protocol change, but you know much better than I do about storage.

Also, what do you think about these numbers? They seem to be reasonable to me, but are they to you too?

---

Next steps

Considering the recent discussion, I'm removing 4 and 5 from the next steps listed above. The next steps thus are:

1. Get feedback on whether the numbers look fine, and the assumptions seem reasonable
2. Implement the protocol change that updates the current costs and adds the less-than-4k costs where applicable
3. Land the PR and rely on release mocknet testing to validate that it does not break too much.
4. In parallel, make sure https://github.com/near/nearcore/pull/11963 lands

[akhi3030]

also CC: @pugachAG

[akhi3030]

this is very good work. I am happy to see that we are moving forward with these cost reductions. Excited for the next steps.

[pugachAG]

@Ekleog-NEAR we also need to consider how storage operations contribute to the chunk state witness size. My intuition says that with memtrie that would mostly determine gas costs, not the latency.

[akhi3030]

@pugachAG: not sure I understand your comment. Gas costs are related to chunk apply latency. If we reduce gas costs but the latency remains high, then we will have undercharging.

[akhi3030]

Just finished a call with @pugachAG to understand their concern above better.

When we access storage, there are two types of costs that we need to consider.

1. the time to actually access the data and return it to the contract. This cost has gone down for reads due to memtrie as seen above.
2. Storage accesses have to be added to the state witness. And there is an upper limit on how bit the state witness can be. Once, we hit this limit, we will no longer include additional transactions in the chunk.

The concern is that if we reduce gas costs based purely on 1. above, then we simply get limited by the state witness size limit. Hence, we should consider both the limits above to decide what the new gas values should be. I agree with this. I don't think it makes sense to reduce gas costs a lot when we will not see significant benefit of that in practice if we keep getting limited by the state witness size.

@Ekleog-NEAR: let's discuss in more detail when you are back from your PTO. I imagine that we should run a few more experiments to figure out how low we can make the gas costs before further reductions will not make sense.

Another point that we discussed in the call is that the current TTN for reading cost is too low. It was set incorrectly when we shipped flat state. So we should actually take this opportunity to see what the correct value should be and if check if it needs to be increased even.

[akhi3030]

@Ekleog-NEAR: I had a conversation with Bowen about this work. A question came up that it seems like we should expect memtrie to help with writes and not with reads. Reads were already optimised due to flat storage.

Finally, there are some other concerns with reducing the gas costs. We should look at what the current compte costs are and instead consider reducing them instead.

[bowenwang1996]

@Ekleog-NEAR when you run the experiments, did you include https://github.com/near/nearcore/commit/0c2374993fc74b57faf2bcdf5c7c73a37e82b75a ? According to @pugachAG this significantly improves memtrie performance.

[Ekleog-NEAR]

The concern is that if we reduce gas costs based purely on 1. above, then we simply get limited by the state witness size limit. Hence, we should consider both the limits above to decide what the new gas values should be. I agree with this. I don't think it makes sense to reduce gas costs a lot when we will not see significant benefit of that in practice if we keep getting limited by the state witness size.

I don't think I agree with this concern. Compute costs and state witness size are two orthogonal limitations, and it just happens that storage reads consume the two. But we have a lot of other operations that consume only compute costs.

So for example, imagine that we have an original workload that does 1 storage read for 1 hash computation. There, it's useless to reduce compute costs by a lot, because we'll very quickly be limited by state witness size. But then a new workload comes in, that does 1 storage read followed by 1M hash computations. If we do not reduce the compute costs, then this workload will be heavily limited by compute costs.

As we don't know what future workloads will be coming on our blockchain, and there's no value in keeping compute costs higher than they should be, I do think that we should reduce them to the lowest value we feel comfortable handling — and increasing them again is not hard anyway.

Another point that we discussed in the call is that the current TTN for reading cost is too low. It was set incorrectly when we shipped flat state. So we should actually take this opportunity to see what the correct value should be and if check if it needs to be increased even.

@pugachAG My understanding was that TTN would only be dealing with values of less than 4k, so would literally never touch the disk trie. Is this understanding wrong?

(I also saw that TTN was set to too low a value when deploying flat storage, but if this understanding is correct then we no longer need to account for the variance there was in flat storage; and thus we can set it to almost the estimator-provided value. Said variance was the reason why TTN costs could at times have been underestimated)

---

@Ekleog-NEAR: I had a conversation with Bowen about this work. A question came up that it seems like we should expect memtrie to help with writes and not with reads. Reads were already optimised due to flat storage.

I probably don't understand this statement, because it seems incompatible with the discussion we had had with @tayfunelmas, summarized above.

Storage reads were already a bit better with flat storage, but memtrie allows us to bypass any disk read, which is a lot of improvement even compared to the only-one-disk-read of flat storage. And on the other hand storage writes still require us to write to all of memtrie, flat storage and on-disk trie, which means that writes are actually a bit slower with memtrie than without.

This understanding is confirmed by the numbers I posted above, so I probably don't understand this sentence.

Finally, there are some other concerns with reducing the gas costs. We should look at what the current compte costs are and instead consider reducing them instead.

The project that was given to me was to reduce gas costs. But as per the first message, I only looked at reducing compute costs for now, and all the "from" numbers given are for compute costs.

This being said, the numbers I suggested changing to are not only lower than our current compute costs, but also lower than our current gas costs. So after this lands and we confirm that it doesn't break anything (ie. after we're sure we won't need to revert the changes in a hurry), we will be able to consider also lowering the gas costs to the new value of the compute costs.

[Ekleog-NEAR]

@Ekleog-NEAR when you run the experiments, did you include https://github.com/near/nearcore/commit/0c2374993fc74b57faf2bcdf5c7c73a37e82b75a ? According to @pugachAG this significantly improves memtrie performance.

I did not; IIRC it was still marked as draft when I ran the experiments. I'll run a new set of experiments and see how things evolve with it.

[Ekleog-NEAR]

Here are the results of the new set of experiments, run on fc87c851c635dd8fdbfc533dcb940125c9355859 so including the changes of 0c23749:

I'm actually not finding much difference between the results with 0c23749 and the results without it: the estimator results are almost always within one standard deviation of the previous results. My guess is that this is because the estimator does not actually exercise the prefetcher, and so the prefetcher is a no-op for it, while it originally brought slowdown to our real-world use case.

TL;DR: No changes with 0c23749, I still recommend the compute costs changes from https://github.com/near/nearcore/issues/11925#issuecomment-2299273181. The speedup from it is probably in parts that had never been measured by the estimator anyway, because it was too much contract-specific.

[Ekleog-NEAR]

Summary of the meeting with @akhi3030, @Longarithm and @pugachAG:

• We don't expect compute costs reductions for the write-specific costs, only for TTN which would then indirectly reduce write costs
• We should run an experiment: add metrics specifically for each cost we're looking to reduce as a histogram, and check that on a mainnet canary node the durations are as we would expect
• Yes TTN only ever touches memtrie, so we don't need to care about the 4k size split
• We currently don't charge TTN for reads because of flat storage, this is something we should fix now that we're significantly reducing compute costs, and because memtrie reintroduces actually touching trie nodes

---

Considering this zulip thread, my assumption on storage_has_key was also correct.

So to sum up the next steps:

1. Split the costs that need to be different for values less than 4k into two costs
2. Reduce the costs that we can simply reduce
3. Add a new ReadTTN cost, with gas cost of 0 and compute cost of 6G, charged on TTN on reads (so that it doesn't charge more gas but only more compute costs)
4. Introduce ReadTTN to the parameter estimator
5. Add histogram metrics to all the costs we're considering changing
6. Run for a while on canary mainnet, check that the metrics are as expected
7. Assuming everything is fine, land
8. (Interestingly, if everything goes fine, we'll have compute costs lower than gas costs here. This being said, we should wait before lowering gas costs, especially because of things like ReadTTN)

With this, this issue has become a bit of a hydra, with lots of new heads popping out. I'd expect the remaining time needed to do it all to be around 3-4 weeks. If we skip steps 5&6, it'd reduce by ~1 week (in exchange for relying on release testing to confirm it works), and probably the same if we skip step 3&4 (and instead charge arbitrarily more on read costs). This being said, this will be my first time adding a new cost, I'll have to hope the documentation we have is still up-to-date, even though it's more than a year old.

For now I'll proceed with doing all the steps, but feel free to let me know if you want me to skip some of them — I think for similar cost changes we usually would have skipped steps 5&6 in particular, because memtrie should have little variance.

[Ekleog-NEAR]

Over the week-end I gave a bit more thought to the suggested cost changes, and I want to change the new TTN cost.

One thing I had initially forgotten was, that (Write)TTN cost must also include the cost for writing the updated node to disk, after updating the leaf value. So we need to plan for more variance in the numbers, because WriteTTN must include disk access time, and disk access has large variance. We don't need to do the same for ReadCachedTrieNode, because we need to write back the node only once anyway.

However, we don't need to handle the full latency that was handled before, because writes are batched at the end of the block processing. So the latency needs to be accounted for only once, and it already is in the Write/Remove costs.

With this in mind, here are my new suggested cost changes. I basically made the 6G of TTN into 20G, in order to account for writeback variance better:

• StorageHasKeyBase: 200,000,000,000 -> 8,000,000,000, without 4k split because checking whether the key exists should AFAIU never require falling back to disk (⚠️ assumption that needs to be confirmed by someone familiar with memtrie)
• StorageHasKeyByte: 30,790,845 -> 9,000,000, without 4k split (⚠️ same assumption)
• StorageReadBase: 200,000,000,000 -> same if the value is 4k or more; 9,000,000,000 if the value is strictly less than 4k
• StorageReadKeyByte: 30,952,533 -> same if the value is 4k or more; 10,000,000 otherwise
• StorageReadValueByte: 5,611,005 -> same if the value is 4k or more; 2,500,000 otherwise
• StorageWrite*: do not change
• StorageRemove*: do not change
• TouchingTrieNode: 110,000,000,000 -> 20,000,000,000, without 4k split (⚠️ same assumption)
• ReadCachedTrieNode: 2,280,000,000 -> 1,400,000,000, we probably do not need the 2x safety margin that we originally had, even though memtrie did not make things much faster the estimator is still around 1,070,000,000 only

[akhi3030]

This looks like a reasonable proposal to me.

[bowenwang1996]

If we plan to make changes to the TrieAccountingCache, should we avoid changing ReadCachedTrieNode for now? cc @akhi3030

[akhi3030]

good point, we will continue discussing and consider this.

[Ekleog-NEAR]

I just noticed an issue with reintroducing ReadTTN for memtrie reads. The details are in this zulip thread; it might result in a change in suggested costs changes.

[Ekleog-NEAR]

I also chatted with Akhi about ReadCachedTrieNode today. Basically, reducing ReadCachedTrieNode is not going to be more complex than not reducing it.

This being said, the gains are not that good. For a random tg receipt, the reduction would lead to a ~2% throughput gain. The gains would be similar for two kaiching receipts.

2% is neither a big nor a small number for the improvement of the throughput of the entire chain.

I can easily leave the cost reduction as-is (so, with the 2% throughput gains) until if/when we make changes to TrieAccountingCache, or remove it to make fewer changes (so, removing the 2% throughput gains).

Regardless of the choice, as we're not touching any gas costs, it'll be just as easy to change TrieAccountingCache later on, and we're not adding any tech debt as it's just a parameter change.

@bowenwang1996 What do you prefer?

[Ekleog-NEAR]

Considering the zulip chat, I ended up removing ReadCachedTrieNode changes altogether for now, until we have more clarity on the next steps

[Ekleog-NEAR]

Also, to copy the next steps from the zulip thread:

1. Reduce gas costs without introducing ReadTTN, as in #12044, for this release
2. Crunch numbers on how much throughput the whole chain would gain if we were to introduce ReadTTN (this will likely be a relatively involved project, as it'll involve figuring out the average depth of read trie nodes, which we don't have good metrics for right now AFAIK)
3. Crunch numbers on how much the operation price of RPC/archival nodes would increase if we were to require memtrie there
4. Assuming we can demonstrate a good enough benefit/cost ratio, implement ReadTTN
5. If not, wait until we have eg. implemented resharding or sharded RPC/archival nodes, and then implement ReadTTN

Currently, 1. is implemented by #12044, and this is just lacking review. As for whether doing steps 2-3 now is worth it right now or I can focus on other important stuff (more performance work), I'll unpop administrative backlog for now to let other people interject opinions, and reconsider after the administrative backlog is handled.

[Ekleog-NEAR]

As part of the evaluation of #12044, I computed a higher bound on the benefits ReadTTN could give us. On the real-world use cases with detailed data available here, the benefits could not exceed 10-20%, as this is the remaining cost of the overestimated read costs.

I'd expect the benefits of just ReadTTN to be around 5-10% of the speed of the whole chain, because most likely, most nodes are at least at depth 12. This being said, the current 6G estimation for ReadTTN is probably also overestimated. If we also re-estimate ReadTTN, we might reach 10-15% of chain throughput improvements