RitaMAllenCA commented 1 year ago

Once we made the necessary updates to store and execute objects using versioned (extended) types, measure the costs of accessing all versions, serializing/deserializing and storing all versions. Any operation that could result in storage bloat or gas cost changes should be measured for differences.

mpapierski commented 1 year ago

Disclaimer

I need to re-run benchmark to measure for 16s and 3.6k CSPR

Example tests for 16s block time and 10k CSPR max gas block

Initial results of benchmarked opcode costs for 16s and 10k CSPR (see above; this is test run, and need to iterate with correct target values).

Diff of the chainspec:

diff --git a/resources/production/chainspec.toml b/resources/production/chainspec.toml
index 9b8d9999a..4b1efc081 100644
--- a/resources/production/chainspec.toml
+++ b/resources/production/chainspec.toml
@@ -155,56 +155,60 @@ gas_per_byte = 630_000

 [wasm.opcode_costs]
 # Bit operations multiplier.
-bit = 300
+bit = 1100
 # Arithmetic add operations multiplier.
-add = 210
+add = 2200
 # Mul operations multiplier.
-mul = 240
+mul = 1360
 # Div operations multiplier.
-div = 320
+div = 1360
 # Memory load operation multiplier.
-load = 2_500
+load = 4300
 # Memory store operation multiplier.
-store = 4_700
+store = 4300
 # Const store operation multiplier.
-const = 110
+const = 2200 # GUESS
 # Local operations multiplier.
-local = 390
+local = 2200
 # Global operations multiplier.
-global = 390
+global = 2800
 # Integer operations multiplier.
-integer_comparison = 250
+integer_comparison = 2000
 # Conversion operations multiplier.
-conversion = 420
+conversion = 1360 # GUESS
 # Unreachable operation multiplier.
-unreachable = 270
+unreachable = 540 # GUESS (whatever?)
 # Nop operation multiplier.
-nop = 200
+nop = 1360 # GUESS
 # Get current memory operation multiplier.
-current_memory = 290
+current_memory = 2800
 # Grow memory cost, per page (64kb).
-grow_memory = 240_000
+grow_memory = 3472985
+
+# validation
+

 # Control flow operations multiplier.
 [wasm.opcode_costs.control_flow]
-block = 440
-loop = 440
-if = 440
-else = 440
-end = 440
-br = 440_000
-br_if = 440_000
-return = 440
-select = 440
-call = 140_000
-call_indirect = 140_000
-drop = 440
+block = 17000
+loop = 17000
+if = 20518
+else = 20518
+end = 20518
+br = 17000
+br_if = 17000
+return = 17000
+select = 17000
+call = 17539
+call_indirect = 20746
+drop = 17000

 [wasm.opcode_costs.control_flow.br_table]
 # Fixed cost per `br_table` opcode
-cost = 440_000
+cost = 17000
 # Size of target labels in the `br_table` opcode will be multiplied by `size_multiplier`
-size_multiplier = 100
+size_multiplier = 200
+

 # Host function declarations are located in smart_contracts/contract/src/ext_ffi.rs
 [wasm.host_function_costs]

Some tests (understandably) fail with different outcomes since some tests will fail with gas limit, etc. Comparison of two sets on the test suite:

total gas a = 32688206971769
total time a = 11026244
total gas b = 30333541869481
total time b = 636
matching entries 2735

where:

a == current prod chain spec
b == patch above
matching entries 2735 - All the exec's in the test suite that have the same outcomes
time is in microseconds, this metric needs improvement

These are only preliminary results, but they show that proposed changes are about ~8% cheaper across all the tests.

Instrumentation

I created PR to merge instrumentation code from the test suite into the code https://github.com/casper-network/casper-node/pull/4278

Data analysis tool

This script compares one execution_result.csv file and other execution_result.csv file: https://gist.github.com/mpapierski/7152edd2eb40782b5b28c28109d660e2

Summary

I need to re-run the benchmarking tool, then run the analysis again.

mpapierski commented 1 year ago

As follow up here's the proposed change for 3.6k CSPR max gas per block, and 16384ms block time. This time, opcode costs we're not "guessed" but instead raw data output of the benchmark was used.

diff --git a/resources/production/chainspec.toml b/resources/production/chainspec.toml
index 5827dcffc..e384fec8f 100644
--- a/resources/production/chainspec.toml
+++ b/resources/production/chainspec.toml
@@ -154,55 +154,54 @@ max_stack_height = 500
 gas_per_byte = 630_000

 [wasm.opcode_costs]
-# Bit operations multiplier.
-bit = 300
+bit = 300 # TODO
 # Arithmetic add operations multiplier.
-add = 210
+add = 413
 # Mul operations multiplier.
-mul = 240
+mul = 398
 # Div operations multiplier.
-div = 320
+div = 398
 # Memory load operation multiplier.
-load = 2_500
+load = 1747
 # Memory store operation multiplier.
-store = 4_700
+store = 1747
 # Const store operation multiplier.
-const = 110
+const = 298
 # Local operations multiplier.
-local = 390
+local = 358
 # Global operations multiplier.
-global = 390
+global = 942
 # Integer operations multiplier.
-integer_comparison = 250
+integer_comparison = 717
 # Conversion operations multiplier.
-conversion = 420
+conversion = 420 # TODO
 # Unreachable operation multiplier.
-unreachable = 270
+unreachable = 270 # TODO
 # Nop operation multiplier.
-nop = 200
+nop = 200 # TODO
 # Get current memory operation multiplier.
-current_memory = 290
+current_memory = 629
 # Grow memory cost, per page (64kb).
-grow_memory = 240_000
+grow_memory = 1223698

 # Control flow operations multiplier.
 [wasm.opcode_costs.control_flow]
-block = 440
-loop = 440
-if = 440
-else = 440
-end = 440
-br = 440_000
-br_if = 440_000
-return = 440
-select = 440
-call = 140_000
-call_indirect = 140_000
-drop = 440
+block = 6401
+loop = 6401
+if = 6401
+else = 6401
+end = 6401
+br = 6401
+br_if = 6401
+return = 6401
+select = 6401
+call = 6401
+call_indirect = 7329
+drop = 6401

 [wasm.opcode_costs.control_flow.br_table]
 # Fixed cost per `br_table` opcode
-cost = 440_000
+cost = 6401
 # Size of target labels in the `br_table` opcode will be multiplied by `size_multiplier`
 size_multiplier = 100

Evaluation of recursive_fibonacci(50) returns gas limit in ~19s. Off by 2616ms (~15%)

Overall gas costs on the test suite:

$ python ../compare.py execution_results_prod.csv execution_results_3600cspr.csv
total gas a = 31528868962549
total time a = 12317971
total gas b = 28579334817437
total time b = 654
matching entries 2695
result: b total gas is 90.64% of a total gas

It also appears initial gas costs are 10% cheaper across all matching exec calls

rafal-ch commented 1 year ago

Another reiteration using the improved measurement tool is needed to go down from the 19s. @mpapierski please provide details in a comment here.

devendran-m commented 1 month ago

Triage update:

Removed from Condor/2.0.0 release.

casper-network / casper-node

evaluate gas costs of extensibility #3923

Disclaimer

Example tests for 16s block time and 10k CSPR max gas block

Instrumentation

Data analysis tool

Summary