threefoldtech / tfchain_graphql

Graphql for TFchain
Apache License 2.0
2 stars 3 forks source link

Use more array programming style to improve firesquid speed #78

Open DylanVerstraete opened 1 year ago

DylanVerstraete commented 1 year ago

For some object we already use "array programming" meaning we collect all creates/updates inside a batch of blocks and save to the database later.

For farms, nodes, contracts, we still save one by one due to relations they have. In a future version, we should convert this to array programming style.

For example, you can read block by block and save entities to the db inbetween or with batch processing you can: read X amount of blocks, save entities to be saved in memory and save to the DB whenever you processed the X amount of blocks => higher performance

renauter commented 11 months ago

More insights from @DylanVerstraete

Should be: we collect all creates/updates inside a batch of blocks and save to the database later Array style programming could be phrased as batch style processing For example, you can read block by block and save entities to the db in between or with batch processing you can: read X amount of blocks, save entities to be saved in memory and save to the DB whenever you processed the X amount of blocks => higher performance Since every "save to the db" opens a connection to the postgres and has some overhead, opening the connection only once and saving an entire bunch of entities is therefor more performant