search

near / near-indexer-for-explorer

Watch NEAR network and store all the data from NEAR blockchain to PostgreSQL database
https://near-indexers.io/docs/projects/near-indexer-for-explorer
GNU General Public License v3.0
123 stars 56 forks source link

Add capability to define data retention time #198

Closed Bexanderthebex closed 2 years ago

Bexanderthebex commented 2 years ago

Currently, we can define where to start indexing (block_height) but it would be awesome if we can also define how long the data can stay in the database so that it won't grow and we can run our database to a considerable size only. Probably something based on the block_timestamp

khorolets commented 2 years ago

Hey there! The initial goal of this project was to store everything from the genesis in the database.

For the tasks we are solving with this project we don't need to delete the old data.

If you need something like that I can suggest to fork the Indexer for Explorer and implement the described feature for yourself.

frol commented 2 years ago

@Bexanderthebex As @khorolets stated, given the name (near-indexer-for-explorer) this project is intended to be used by NEAR Explorer, which needs the whole history, so the proposed feature does not fit our needs. Feel free to fork the project and/or submit a PR. If the PR would look reasonable, we might merge it, but realistically, I feel that would create a maintenance burden on our side, which is not helping us, so I would recommend you go with a fork.

frol commented 2 years ago

P.S. it is not as simple as just deleting old blocks since accounts and access_keys (and some other tables) have references to the receipts that created or deleted those accounts/keys, and thus you would need to take a closer look into those relations and drop the foreign key constraint.

Bexanderthebex commented 2 years ago

@frol I agree with what @khorolets said. I think I'll just create a fork or we will just configure a non-archival node that will store the last 7 days worth of transactions.