search

ziyadedher / evm-bench

🚀🪑 evm-bench is a suite of Ethereum Virtual Machine stress tests and benchmarks.
GNU General Public License v3.0
201 stars 21 forks source link

Bump to revm 3.0 #153

Open gakonst opened 1 year ago

gakonst commented 1 year ago

Dragan released a new Revm version with some perf improvements, https://crates.io/crates/revm

gakonst commented 1 year ago

hi @ziyadedher wonder if you have some capacity for this? revm has gotten some improvements lately and would love to get an update in the table

ziyadedher commented 1 year ago

Thanks for the bump! Will do this today. I'm on the road and don't have my main computer, so I might just update all the results on my laptop. Either way, relative results should still be stable.

ziyadedher commented 1 year ago

Benchmarks are running on my laptop now, couldn't get pypy to compile properly on my Mac (at least not for Python 3.9), so just gonna do without for now. Will post a PR once the benchmark results come through.

ziyadedher commented 1 year ago

I'm seeing some surprising results on my MacBook (maybe ignore ethereumjs since my node-js runner is probably very optimized for work):

evmone revm pyrevm geth py-evm.cpython ethereumjs
sum 69.2ms 100.4ms 218ms 231.4ms 21.7272s 31.3376s
relative 1.000x 1.451x 3.150x 3.344x 313.977x 452.855x
erc20.approval-transfer 7.4ms 10.2ms 21.8ms 17.4ms 1.374s 1.8832s
erc20.mint 5.2ms 6.2ms 16.8ms 18.4ms 1.2822s 2.8656s
erc20.transfer 8.6ms 12ms 24.4ms 26ms 1.8158s 3.3676s
snailtracer 45ms 67ms 143ms 157ms 15.455s 21.592s
ten-thousand-hashes 3ms 5ms 12ms 12.6ms 1.8002s 1.6292s

I want to re-run these benchmarks when I'm back at my computer as a sanity check, but digging a bit deeper I see this:

revm.old revm.new
sum 94.2ms 109.3ms
relative 1.000x 1.160x
erc20.approval-transfer 10.6ms 11.4ms
erc20.mint 6.2ms 7.4ms
erc20.transfer 11.4ms 13ms
snailtracer 61.6ms 72.5ms
ten-thousand-hashes 4.4ms 5ms

Where revm.old is using pre-#212 revm-interpreter and revm.new is using latest at 1.1.1.

In the meantime, @rakita do you have any thoughts on a potential regression here? I think I'm actually seeing a slight regression from the old "release" of revm-interpreter to the latest one (1.1.1). Is the way the runner is setup still optimal?

@gakonst does this surprise you given how you feel Foundry or Reth perf has been?

gakonst commented 1 year ago

Thx we'll investigate!

ziyadedher commented 1 year ago

Sounds good! Please keep me in the loop, I definitely wanna make sure that this benchmark suite is not misleading.

rakita commented 1 year ago

This is expected, we removed one optimization (gas block) for more flexibility in lib usage, I got a little bit less performance degradation as a lot of jump predictions did they work, but i think this is run in docker so it is more effected.

I would say run it a few more times (2/3) just to confirm it is similar results and that would be it.