checkpoint-labs / checkpoint

Checkpoint is a library for indexing data of Starknet contracts.
https://checkpoint.box
MIT License
55 stars 22 forks source link

fix: Maximum call stack size exceeded with array.push #302

Closed shardAstronaut closed 6 days ago

shardAstronaut commented 6 days ago

Description

When indexing contracts with a big number of events the array.push method will throw an error. When you use spread operator all items of the source array are stored in the stack as arguments list, so having a large number of items (~ > 100K) will cause the this stack size exceed. The most simple work-around is to manually push all items one by one. https://stackoverflow.com/questions/61740599/rangeerror-maximum-call-stack-size-exceeded-with-array-push

shardAstronaut commented 6 days ago

Do you know if this issue still happens in your case if we use concat instead of push in a loop?

I wonder if concat:

  1. solves the issue
  2. is faster
  3. Yes it solves the issue
  4. Looks like concat is faster than push based on this discussion https://stackoverflow.com/questions/48865710/spread-operator-vs-array-concat. I can update the PR to use concat it's cleaner, wdyt?
Sekhmet commented 6 days ago

Let's do concat then, thanks!

shardAstronaut commented 6 days ago

Do you know what changed in Starknet sequancer to start getting this error now? I was indexing the ETH contract for more than 4 months and never get this error.

Sekhmet commented 6 days ago

Do you know what changed in Starknet sequancer to start getting this error now? I was indexing the ETH contract for more than 4 months and never get this error.

Most likely it's caused by the fact that we changed the way we fetch events to find blocks we need to index (before we fetched all events, but only for 100 blocks at the time, now we fetch 1000 blocks or more at the time, but filtering events that only come from contracts we track). Generally it causes way faster indexing, but looks like there is room for improvements for certain contracts.

Sekhmet commented 6 days ago

Thank you!