https://rpc.ankr.com/gnosis has a batch limit

[hboon]

https://github.com/AlphaWallet/alpha-wallet-ios/pull/5024

[JamesSmartCell]

@seabornlee note that the current log system may already deal with this; because each chain can fetch different amounts of logs we built a generic system that automatically finds how many logs can be pulled.

But, unit tests for this system should be a must.

[hboon]

@JamesSmartCell how is the window size automatically detected? Sounds useful.

[JamesSmartCell]

@JamesSmartCell how is the window size automatically detected? Sounds useful.

I think I wrote it up. I tested it on all the networks like Mainnet, Mumbai etc. It works by ratcheting the fetched logs; mainly because you have no idea how many logs will be in a given window, so the batch limit is kind of an irrelevant aside.

EG if a fetch from block 1 to current block (eg 60064403) for transfer event logs related to me on an NFT, there might only be say 4 receive logs and 2 send logs. Then, no matter how small the batch limit is, it won't affect us.

I'll share the heuristic I used (Although I thought I already shared it).

It works on all the chains I tested it with (using a high volume account, I like to test on a major ERC20 like USDC as you get huge tx volumes to really thrash the algorithm) but I'd feel a lot better with a unit test.

With this way, you never need to worry about the chain limit, as it's automatically accounted for.

[hboon]

@JamesSmartCell There are 4 related limits here, optionally enforced by RPC nodes:

A. The block number range for a single eth_getLogs call B. The number of events that can be returned for a eth_getLogs call C. The number of JSON/RPC calls in a single batch JSON/RPC call (e.g we make N eth_getBalance, one for each wallet all in a single batch call) D. Specialization of (C): our Gnosis node apparently also limits the number of eth_getLogs within each batch

This issue (#2736) is refering to (D).

2735 refers to (A).

I think I wrote it up. I tested it on all the networks like Mainnet, Mumbai etc. It works by ratcheting the fetched logs; mainly because you have no idea how many logs will be in a given window, so the batch limit is kind of an irrelevant aside.

Which among A-D does this address?

[hboon]

cc @seabornlee who might be interested in following since https://github.com/AlphaWallet/alpha-wallet-android/issues/2735#issuecomment-1193495214

[JamesSmartCell]

@JamesSmartCell There are 4 related limits here, optionally enforced by RPC nodes:

A. The block number range for a single eth_getLogs call B. The number of events that can be returned for a eth_getLogs call C. The number of JSON/RPC calls in a single batch JSON/RPC call (e.g we make N eth_getBalance, one for each wallet all in a single batch call) D. Specialization of (C): our Gnosis node apparently also limits the number of eth_getLogs within each batch

This issue (#2736) is refering to (D).

2735 refers to (A).

I think I wrote it up. I tested it on all the networks like Mainnet, Mumbai etc. It works by ratcheting the fetched logs; mainly because you have no idea how many logs will be in a given window, so the batch limit is kind of an irrelevant aside.

Which among A-D does this address?

A. doesn't appear to be enforced on any of the chains (that I tested against). Providing there are fewer than the event limit events, all can be returned by asking for block 1 -> currentBlock. Do any of the chains we target enforce this limit? B. Is what the heuristic works with. Eg trying to retrieve all events on DAI contract that were generated from a very active account with 100,000's of calls from early history. C and D are not relevant to Android: I wasn't aware we can batch call eventlogs. Is there some documentation available or raw JSON examples to illustrate this?

[hboon]

A. doesn't appear to be enforced on any of the chains (that I tested against). Providing there are fewer than the event limit events, all can be returned by asking for block 1 -> currentBlock. Do any of the chains we target enforce this limit?

A. Pretty sure it would matter unless we somehow already set (by default maybe?) a small enough limit. It's this limit:

myContract.getPastEvents('Transfer', {
  filter: {
    from: "0xb8ae138Dd157811AD7f2Ed6D839614E12Fc17c68"
  }
  fromBlock: 0,
  toBlock: 3000
}, function(error, events){ console.log(error, events); })

This block number range is referring to 3000 - 0 in the snippet above.

B. Is what the heuristic works with. Eg trying to retrieve all events on DAI contract that were generated from a very active account with 100,000's of calls from early history.

Right. I think in practice, we wouldn't hit this limit for wallets unless we watch a whale or exchange, or maybe some niche contracts on a chain with very cheap gas?

C and D are not relevant to Android: I wasn't aware we can batch call eventlogs. Is there some documentation available or raw JSON examples to illustrate this?

It's part of JSON/RPC (not Ethereum or EVMs) https://www.jsonrpc.org/specification#batch

"normal"/un-batched:

curl -H 'accept: application/json' -H 'content-type: application/json' -H 'user-agent: AlphaWallet/395 CFNetwork/1237 Darwin/20.5.0' -H 'accept-language: en-us' --data-binary '{"method":"eth_getTransactionCount","params":["0xbc8dAfeacA658Ae0857C80D8Aa6dE4D487577c63", "pending"],"id":1,"jsonrpc":"2.0"}' --compressed "https://mainnet.infura.io/v3/ad6d834b7a1e4d03a7fde92020616149"

{"jsonrpc":"2.0","id":1,"result":"0x8c2"}

Batched (N = 2):

curl -H 'accept: application/json' -H 'content-type: application/json' -H 'user-agent: AlphaWallet/395 CFNetwork/1237 Darwin/20.5.0' -H 'accept-language: en-us' --data-binary '[{"method":"eth_getTransactionCount","params":["0xbc8dAfeacA658Ae0857C80D8Aa6dE4D487577c63", "pending"],"id":1,"jsonrpc":"2.0"},{"method":"eth_getTransactionCount","params":["0xbc8dAfeacA658Ae0857C80D8Aa6dE4D487577c63", "pending"],"id":1,"jsonrpc":"2.0"}]' --compressed "https://mainnet.infura.io/v3/ad6d834b7a1e4d03a7fde92020616149"

[
  {"jsonrpc":"2.0","id":1,"result":"0x8c2"},
  {"jsonrpc":"2.0","id":1,"result":"0x8c2"}
]

The difference is just changing the JSON payload from a single dictionary to an array of dictionaries.

[hboon]

A. doesn't appear to be enforced on any of the chains (that I tested against). Providing there are fewer than the event limit events, all can be returned by asking for block 1 -> currentBlock. Do any of the chains we target enforce this limit?

The ones we discovered for iOS: https://github.com/AlphaWallet/alpha-wallet-ios/blob/4d41f17c1e2ea44ad733ee3eb90be33fa7b0cbe6/AlphaWallet/Settings/Types/RPCServers.swift#L868

[JamesSmartCell]

A. doesn't appear to be enforced on any of the chains (that I tested against). Providing there are fewer than the event limit events, all can be returned by asking for block 1 -> currentBlock. Do any of the chains we target enforce this limit?

The ones we discovered for iOS: https://github.com/AlphaWallet/alpha-wallet-ios/blob/4d41f17c1e2ea44ad733ee3eb90be33fa7b0cbe6/AlphaWallet/Settings/Types/RPCServers.swift#L868

Are these limits for the number of results or the range? Only I tested most of those and I was able to get a return range from block 1 to current, providing the number of events was within the result limit (eg 3500).

I'll post a postman here that shows you can do this with any of these chains (or at least, it'll show we can't and with which ones):

[hboon]

https://github.com/AlphaWallet/alpha-wallet-ios/blob/4d41f17c1e2ea44ad733ee3eb90be33fa7b0cbe6/AlphaWallet/Settings/Types/RPCServers.swift#L868 is about range.

Example, 0 - latest:

curl -H 'accept: application/json' -H 'content-type: application/json' -H 'user-agent: AlphaWallet/395 CFNetwork/1237 Darwin/20.5.0' -H 'accept-language: en-us' --data-binary '[{"jsonrpc":"2.0","method":"eth_getLogs","id":1,"params":[{"fromBlock": "0x0", "toBlock":"latest","topics":[null,null,null,"0x00000000000000000000000080cc263cb3fa1be2aec748b3811261c1e2a1ba8d"]}]}]' --compressed 'https://bsc-dataseed.binance.org'

result, range exceeded:

[{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"exceed maximum block range: 5000"}}]

0 - 5000 (0x1388):

curl -H 'accept: application/json' -H 'content-type: application/json' -H 'user-agent: AlphaWallet/395 CFNetwork/1237 Darwin/20.5.0' -H 'accept-language: en-us' --data-binary '[{"jsonrpc":"2.0","method":"eth_getLogs","id":1,"params":[{"fromBlock": "0x0", "toBlock":"0x1388","topics":[null,null,null,"0x00000000000000000000000080cc263cb3fa1be2aec748b3811261c1e2a1ba8d"]}]}]' --compressed 'https://bsc-dataseed.binance.org'

result, OK:

[{"jsonrpc":"2.0","id":1,"result":[]}]

0 - 5001 (0x1389):

curl -H 'accept: application/json' -H 'content-type: application/json' -H 'user-agent: AlphaWallet/395 CFNetwork/1237 Darwin/20.5.0' -H 'accept-language: en-us' --data-binary '[{"jsonrpc":"2.0","method":"eth_getLogs","id":1,"params":[{"fromBlock": "0x0", "toBlock":"0x1389","topics":[null,null,null,"0x00000000000000000000000080cc263cb3fa1be2aec748b3811261c1e2a1ba8d"]}]}]' --compressed 'https://bsc-dataseed.binance.org'

result, range exceeded:

[{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"exceed maximum block range: 5000"}}]

[JamesSmartCell]

I did extensive tests with range vs event count, however I didn't test on eg BSC. Which chains specifically did you encounter the range issue? Most chains eg Polygon, Klaytn support full range checks.

[hboon]

Which chains specifically did you encounter the range issue?

https://github.com/AlphaWallet/alpha-wallet-ios/blob/4d41f17c1e2ea44ad733ee3eb90be33fa7b0cbe6/AlphaWallet/Settings/Types/RPCServers.swift#L868

[JamesSmartCell]

I am sure I got a range from 1 -> latest on Polygon at least. Will need to re-test to double check