search

filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.82k stars 1.25k forks source link

CLI for internal consistency testing of RPC endpoints #12412

Open BigLep opened 3 weeks ago

BigLep commented 3 weeks ago

Done Criteria

There exists a CLI that can take an RPC endpoint to then make RPC calls so it can then make internal consistency checks against a live endpoint. It doesn’t assume the endpoint has fixed state. Instead it will have logic like “because I got response X for API Y, I expect to see Z in response to other related APIs”.

Timing measurements are also collected and emitted.

We won’t be doing exhaustive API coverage. Rather, this is a “beachhead”. For user-provided failure cases we’ll write a reproducible case in this tool so we can reliably follow the same steps (rather than relying on copy-pasting curl commands), test other endpoints, and confirm that any fixes no longer manifests the issue.

Why Important

This is the beginning of having more of a mechanism to better debug, catch, and report on RPC correctness and performance. It doesn't scale to require "best intentions" to test certain scenarios or to manually try and follow various repo steps across various endpoints.

User/Customer

  1. Maintainers trying to improve ETH RPC correctness and performance.
  2. RPC operators who want to see how their endpoint compares against other endpoints for some standard callpaths.

Notes

  1. This is a new repo outside of Lotus. I'm creating this issue here for now, but it can be moved when the new repo has been created.
  2. It’s TBD whether this will be written in Go or a language like JS/TS that end clients actually use.
  3. This may develop into creating a public dashboard comparing RPC providers across various dimensions.
  4. There is some synergy/overlap here with https://benchmark-rpc.fil.builders/ and https://github.com/ChainSafe/filecoin-common-node-api
BigLep commented 1 week ago

2024-09-04 update: @rvagg is active on this. He has generated a JS/TS client from an IPLD schema for the Lotus Common Node API and ETH RPC apis. He is calling Lotus APIs to get the data directly from the blockstore (bypassing sqllite) and will then call eth_getLogs to confirm consistent state. He's aiming to complete by end of week. That is the checkpoint we'll use to then determine next steps including:

  1. structuring/repo
  2. what additional asserts/consistency-checks we should do