search

ethereum / hive

Ethereum end-to-end test harness
GNU General Public License v3.0
399 stars 449 forks source link

rpc-compat: Run `speccheck` on non-deterministic tests #640

Open lightclient opened 2 years ago

lightclient commented 2 years ago

Some methods are not deterministic across clients. For example, eth_gasPrice returns as gasPrice in wei that users could use for a transaction to be included quickly. This is subjective based on client so a static test case doesn't make sense. However, we know what the shape the request and response should look like. So it would make sense to make the request anyways and verify the round trip against the spec.

I've written a tool speccheck which more-or-less does this. However it doesn't yet have the ability to verify round-trips that aren't written to file in the format of the test fixtures. So it would need to be librarified to a degree and then imported into rpc-compat and run against certain non-deterministic endpoints (maybe even all of them!).

fjl commented 8 months ago

I'm adding a speconly flag in ethereum/execution-apis#520 so we can at least mark the tests where this kind of checking should occur.

lightclient commented 8 months ago

This is a really nice idea, I am happy to begin using it.

fjl commented 8 months ago

I'm working on it right now!

fjl commented 8 months ago

Parsing of the flag is now implemented in the simulator, but it's unclear to me how to proceed with adding speccheck. I don't want to depend on rpctestgen from the simulator, it defeats the purpose of having the generator and test output separated.