indexsupply / shovel

An Ethereum to Postgres indexer
https://indexsupply.com/shovel
MIT License
172 stars 22 forks source link

shovel: remove batch cache #223

Closed ryandotsmith closed 9 months ago

ryandotsmith commented 9 months ago

This may be a classic case of a failed attempt to outsmart the compiler. In a prior version of the program, I was holding on to the task's blocks via batch so that I could reduce the amount of txs/logs/topics that needed to be allocated.

However, in recent benchmarking, and in dealing with a nasty memory leak found in and around the jrpc2 client, I found that task.batch was preventing the GC from reducing memory. Once the batch was made ephemeral, the program got faster and the memory usage was reduced.

Consequently, we no longer need to think about resetting data structures. We create new ones. When in GC Rome...