Indexer makes one fetch for full block range

[plotchy]

The block range im concerned about gathering events for is ~600K wide.

the resulting rpc call is for eth_getLogs for the full range, which always will time out for my use case. Can we configure step size somehow?

The example where the end block is only 100 blocks ahead works perfectly

name: kamicsv
project_type: rust
networks:
- name: yominet
  chain_id: 5264468217
  rpc: https://yominet.rpc.caldera.xyz/http
storage:
  csv:
    enabled: true
    path: ./generated_csv
contracts:
- name: World
  details:
  - network: yominet
    address: 0x441e13a25caecad50028e7623a39b91a507bca02
    start_block: '1077466'
    # end_block: '1077566'
    # end_block: '1650276'
  abi: ./abis/World.abi.json
  include_events:
  - ComponentValueSet

20 July - 14:33:20.921676 TRACE rpc{method="eth_getLogs" params="[{\"fromBlock\":\"0x10713e\",\"toBlock\":\"0x193348\",\"address\":\"0x441e13a25caecad50028e7623a39b91a507bca02\",\"topics\":[\"0x6ac31c38682e0128240cf68316d7ae751020d8f74c614e2a30278afcec8a6073\"]}]"}: take? ("https", yominet.rpc.caldera.xyz): expiration = Some(90s)

[joshstevens19]

Hey so to note mostly all the main RPC providers bring back a response if range to large and then from there you can index the most optimal amount of logs each time

code: https://github.com/joshstevens19/rindexer/blob/master/core/src/indexer/fetch_logs.rs#L503 info: https://rindexer.xyz/docs/references/rpc-node-providers#rpc-node-providers

I was going to add a max_block_size but i didn't in the end, the RPC provider your using does do the above? scanning blocks by blocks is slower then doing what the above RPC providers do. I can for sure add this on the contract > details level so you can config max_block_size per contract or would you expect it on the network level where you config the rpc node?

[joshstevens19]

I feel the best place here is to add it on the network level as it tends to be network RPC globally these limits

[plotchy]

unfortunately this rpc provider is the only one available for the network and doesnt provide an optimal range. The RPC docs say that only 1000 events can be returned, and any over that then responds with an error.

So generally I've been indexing 20 blocks per request...😭 It's slow but it's consistent.

[plotchy]

I feel the best place here is to add it on the network level as it tends to be network RPC globally these limits

yes i agree

[joshstevens19]

ha ha makes sense!

i need to add this anyway for something else I'm doing for rindexer so il switch to do this tonight for us, i think its small thing as we have all the code anyway just on the Option need to pick it from the manifest yaml and expose a field to define it

[plotchy]

awesome!

[joshstevens19]

https://github.com/joshstevens19/rindexer/commit/8ff14c102962606da4318eebb2faf7f9fbdb34c2 https://rindexer.xyz/docs/start-building/yaml-config/networks#max_block_range

will deploy in next 1 hour

[joshstevens19]

https://github.com/joshstevens19/rindexer/pull/59

[joshstevens19]

rindexerup - version 0.2.0 👍

[plotchy]

this works perfectly. and dang is it fast! about 3x faster than my crappy python indexer. just switched over! ty

[joshstevens19]

That’s what we like the hear! 🔥