austinabell
commented
4 years ago So just to clarify before I try to give details about how benchmarking can be done, the main difference between Ethereum and Ethermint to benchmark is the relative cost of state commits happening at the end of the block. As in Ethereum, all transactions build a cache of changes to be committed at the end of the block and the difference between Ethereum and Ethermint is how those changes are stored.
Where the state committed is in this function https://github.com/ChainSafe/ethermint/blob/c99d5cf6c5ca0b4e667fc55d72292b5eef04a7b2/x/evm/types/statedb.go#L340 but that whole file can be compared with and benchmarked against Ethereum's equivalent StateDB struct implementation.
If you want to try to single out the state committal, the code set up by Alexander Bez from Cosmos https://github.com/ChainSafe/ethermint/blob/development/importer/importer_test.go might lead you in the right direction as the main difference here is just the state commit at the end of the block. He also made a commit on a branch to add memory usage to this block importer test: https://github.com/ChainSafe/ethermint/commit/f8d4ae4bd66c7605691577d7090497c9e69c0267 if that helps.
Another option for benchmarking is to flood the web3 RPC api with raw transactions on both Ethereum and Ethermint nodes and compare the amount of transactions they can process (per second/ per block or whatever) which might be a more accurate way of benchmarking the total throughput in a simulated system, but probably won't help out much with relative gas costs unless you add useful metrics as well as this.
Also just want to make clear that the database write and read costs is determined internally by the Cosmos version being used. Cosmos (or more specifically Tendermint) uses an IAVL+ tree vs Ethereum's Merkle Patricia Trie (and I know Cosmos was looking into updating before since it was becoming a bottleneck). As for the underlying KV store, Ethereum uses leveldb and so does Tendermint as a default (but just to note they do support swapping the underlying db options: https://github.com/tendermint/tm-db/blob/master/db.go). There is additional functionality added to the state transition to merge the two systems as they need to be, just saying it's important to target what you are actually trying to benchmark.
You could also modify the importer test or create a new one to process transactions through the EVM module. I believe the plan they vaguely had was to have the EVM module and Ethermint/web3 api (just trying to swap the underlying store and consensus with Tendermint) but those are all connected as one where transactions through the web3 API are routed the same as Cosmos standard transactions through the EVM module. If you want to benchmark the EVM module accurately, you also need to take into account the AnteHandler in Ethermint. In the Cosmos ecosystem a transaction processing is split into two parts, the verification in the AnteHandler and the state transition at the end of the block, where Ethereum handles everything in one pass. The verification reads and computation in this AnteHandler probably should not be ignored.
To summarize what it seems like to me for benchmarking the options include (but not limited to these or using specifically one):
- Adding metrics to state db operations
- Benchmarking importing blocks from Ethereum using existing logic in
importer_test.goto compare the write/read cost difference against Ethereum and be able to estimate the relative gas cost changes needed - Creating new importer test which more accurately maps the transaction flow of Ethermint/Cosmos and use that to benchmark
- Flooding Ethermint web3 API and comparing against the same benchmark on an individual Ethereum node on fresh state. (There may be existing tooling to do this and would benefit by also adding metrics internally to know where bottlenecks may be)
github-actions[bot]
We should bencmark opcode costs between Geth and Ethermint.
We want to understand the ram and CPU usage cause we need to know what machines people can run Aragon Chain on. First part is CPU and RAM, @austinabell will write up on how to do a high load test. We could write a script to aggressively send lots of tx's and contract interactions for stress testing.
Part 2: benchmarking the opcodes themselves. This will be a bit more challenging, we have to actually analyze the time spent per opcode