paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
3.95k stars 1.19k forks source link

Add support for Otterscan JSON-RPC API extensions #3726

Open ZePedroResende opened 1 year ago

ZePedroResende commented 1 year ago

Describe the feature

Otterscan is an Ethereum block explorer designed to be run locally with an archive node companion. It's more private compared to other solutions because it allows you to query your node, avoiding sharing sensitive data with third parties. This is achieved without an external indexer through custom RPC methods that currently are only available in Erigon. The addition of this extension (even if behind a feature flag) would allow the community to run Otterscan with a different client other than Erigon. Users would have access to useful information not available on the standard RPC endpoints, without an external indexer, which can be very helpful for data-intensive applications.

### Endpoint checklist
- [x] `ots_getApiLevel`
- [x] `ots_getInternalOperations`
- [x] `ots_hasCode`
- [x] `ots_getTransactionError`
- [x] `ots_traceTransaction`
- [x] `ots_getBlockDetails`
- [x] `ots_getBlockDetailsByHash`
- [x] `ots_getBlockTransactions`
- [ ] `ots_searchTransactionsBefore`
- [ ] `ots_searchTransactionsAfter`
- [ ] `ots_getTransactionBySenderAndNonce`
- [ ] https://github.com/paradigmxyz/reth/issues/8638

Additional context

No response

naps62 commented 1 year ago

@gakonst as discussed, can we get this assigned to @ZePedroResende ? I'll be working on this with him

I was thinking a good first step would be to get an initial endpoint done as a draft, mainly to start discussing architecture. I imagine some input will be required, so I want to optimize everyone's time :)

gakonst commented 1 year ago

SGTM! Let us know how to proceed here.

mattsse commented 1 year ago

cool, some pointers: the way it's structure is that we define a trait per namespace with all the functions, like:

https://github.com/paradigmxyz/reth/blob/94ba83f6353cc8d8e25c6acc118b8f6e90145511/crates/rpc/rpc-api/src/web3.rs#L4-L16

and then add a type that implements this like:

https://github.com/paradigmxyz/reth/blob/94ba83f6353cc8d8e25c6acc118b8f6e90145511/crates/rpc/rpc/src/web3.rs#L23-L38

naps62 commented 1 year ago

@mattsse thanks, that helped a lot already

what's your preference regarding how to manage the work here? a single large branch where we built the feature and merge as a whole? merging small incremental changes, disabled through a feature flag?

mattsse commented 1 year ago

incremental would be ideal, for example

  1. trait bindings
  2. impl (can be done in multiple steps)
  3. integration in rpc-builder+cli
naps62 commented 1 year ago

Bringing back @gakonst 's comment from the PR, feels more on-topic here:

My main Q/concern here is: What new tables are needed to support these APIs? Could we scope them very clearly, as well as understand 1) what stages (if any) we need to add to generate these new tables, 2) how much DB overhead they incur? My main Q/concern here is: What new tables are needed to support these APIs?

makes sense. We went this route because of mattsse's suggestion, but I do admit we still need to learn more about that. also because we're still learning the codebase here, so any input is much appreciated

I was getting more familiar with reth storage because of this topic yesterday. Here's my thoughts so far (I'll let @ZePedroResende correct/complete me):

Maybe some of these can leverage existing tables that I overlooked? I also assume we'll at the very least want to feature-gate these features to avoid the extra overhead unless explicitly needed?

gakonst commented 1 year ago

I also assume we'll at the very least want to feature-gate these features to avoid the extra overhead unless explicitly needed?

100% - for sure.

cc @joshieDo as these are all indices and it might overlap with the future etl work

ZePedroResende commented 1 year ago

Endpoint checklist

onbjerg commented 1 year ago

@ZePedroResende moved the tasklist to the main issue

0xAlcibiades commented 7 months ago

This still seems to be missing a few parts including the erigon namespace for otterscan support to work. I see this was recently added to foundry and wonder if there'd be interest to bring things up to speed on this side @Rjected @Evalir @ZePedroResende

0xAlcibiades commented 7 months ago

Relates to: https://github.com/foundry-rs/foundry/pull/5414

wakamex commented 5 months ago

without ots_getContractCreator I can't use the dope new Ape feature which stores contract creation metadata: https://github.com/ApeWorX/ape/pull/2001

wtdcode commented 4 months ago

Anyone working on the ots_searchTransactionsBefore, ots_searchTransactionsAfter and ots_getTransactionBySenderAndNonce? I'm going to give it a try.

Rjected commented 4 months ago

Anyone working on the ots_searchTransactionsBefore, ots_searchTransactionsAfter and ots_getTransactionBySenderAndNonce? I'm going to give it a try.

hey @wtdcode ! w.r.t. new ots endpoints, we want to prioritize those that do not require adding extra indexing to reth, for example ots_getContractCreator looks like it doesn't necessarily need an extra index. I don't know much about ots_searchTransactionsBefore / ots_searchTransactionsAfter / ots_getTransactionBySenderAndNonce, so the first step there would be taking a look at other implementations of those endpoints, and checking if they can be done without extra indices on the reth end. If they do not, we can then help guide an implementation much more easily. Does that make sense?