near / near-indexer-for-explorer

Watch NEAR network and store all the data from NEAR blockchain to PostgreSQL database
https://near-indexers.io/docs/projects/near-indexer-for-explorer
GNU General Public License v3.0
123 stars 56 forks source link

Split batch inserts to pieces of the known size #231

Open telezhnaya opened 2 years ago

telezhnaya commented 2 years ago

During #224, we discussed that it's better to put the data to the DB by pieces (e.g., 100 or 500 rows at a time), as we are doing it for receipts and/or transactions implementation. I've checked: we don't do that anywhere except storing genesis https://github.com/near/near-indexer-for-explorer/blob/master/src/db_adapters/genesis.rs#L63

I think it's a really good idea, especially with the increased load on the DB.

telezhnaya commented 2 years ago

https://explorer.mainnet.near.org/blocks/3N1J6F2V5LUmoAiHdQJVM8Lo4DSSmXyF8HpJKM22YWjo 85 tx, 129 receipts

select * from account_changes where account_changes.changed_in_block_hash = '3N1J6F2V5LUmoAiHdQJVM8Lo4DSSmXyF8HpJKM22YWjo';

254 rows with one batch insert

not that bad, actually, but it's better to fix it before it provides us with the problems