near / near-indexer-for-explorer

Watch NEAR network and store all the data from NEAR blockchain to PostgreSQL database
https://near-indexers.io/docs/projects/near-indexer-for-explorer
GNU General Public License v3.0
124 stars 56 forks source link

feat: introduced an easy way to sort account_changes in their correct historical order (changed_in_block_timestamp + index_in_block) #142

Closed telezhnaya closed 3 years ago

telezhnaya commented 3 years ago

Partially resolves #77 (this PR introduces an easy way to sort the account changing events: ORDER BY changed_in_block_timestamp, index_in_block)

I am still waiting for the server updates, want to test it on the production-like env, that's why I marked it as a draft.

We should apply the migration by pieces, otherwise, it will take too long to update existing lines.

frol commented 3 years ago

@telezhnaya how long does it take to apply the migration?

telezhnaya commented 3 years ago

@frol the honest answer is "too long anyway". 3 minutes if everything was already updated before. More than one hour to apply it till 31 December 2020 at once (killed this process because it's insane anyway).

I think the best idea is to comment UPDATE statement in up.sql so that we will never apply this migration fully. The comment will be used just to know the idea of how it should be done. In the real world, we should apply it in pieces. I suggest creating a script that updates the data splitting it by timestamp.

frol commented 3 years ago

I think the best idea is to comment UPDATE statement in up.sql so that we will never apply this migration fully. The comment will be used just to know the idea of how it should be done. In the real world, we should apply it in pieces. I suggest creating a script that updates the data splitting it by timestamp.

This sounds like a plan :+1:

khorolets commented 3 years ago

@telezhnaya last time we had an issue with "too timeconsuming" migration I've come up with a script and left it for the history here https://github.com/near/near-indexer-for-explorer/pull/99#issuecomment-838871700 you might find it handy. Let me know if you need my help.

frol commented 3 years ago

I just remembered that we may also need an index over the (changed_in_block_timestamp, index_in_block) to speed up sorting.