near / near-indexer-for-explorer

Watch NEAR network and store all the data from NEAR blockchain to PostgreSQL database
https://near-indexers.io/docs/projects/near-indexer-for-explorer
GNU General Public License v3.0
123 stars 56 forks source link

chore: Reindex `access_keys` #321

Closed morgsmccauley closed 1 year ago

morgsmccauley commented 1 year ago

Creating a temporary branch which modifies indexer so that it only writes to access_keys_reindexed. This is so we can reindex the entire access_keys table so that the data is not affected by the bug described in #304.

As I am not that familiar with the codebase, I wanted to create this PR to get a second pair of eyes and ensure I am on the right track. I'll pull down this branch on the testnet/mainnet machines to build the binary, rather than patching it on the machine directly.

morgsmccauley commented 1 year ago

Here's the steps I'm planning to take to reindex the table. Am I on the right track here?

  1. Create a new temporary table
    CREATE TABLE public.access_keys_reindexed (like public.access_keys);
  2. Check out this branch and build binary
    cd near-indexer-for-explorer
    git checkout chore/access-keys-reindexed
    cargo build --release
  3. Update indexer.service so that it syncs from genesis
    - ExecStart=/home/ubuntu/indexer run --stream-while-syncing sync-from-interruption --delta 500
    + ExecStart=/home/ubuntu/near-indexer-for-explorer/target/release/indexer-explorer run --store-genesis sync-from-block --height 9820210
  4. Restart indexer service
    systemctl restart indexer
  5. Once caught up, stop the service
    systemctl stop indexer
  6. Add indexes and foreign-key constraints to new table, then swap
    
    START TRANSACTION;

ALTER TABLE ONLY public.access_keys DROP CONSTRAINT access_keys_pk; ALTER TABLE ONLY public.access_keys_reindexed ADD CONSTRAINT access_keys_pk PRIMARY KEY (public_key, account_id);

DROP INDEX access_keys_account_id_idx; CREATE INDEX access_keys_account_id_idx ON public.access_keys_reindexed USING btree (account_id); DROP INDEX access_keys_last_update_block_height_idx; CREATE INDEX access_keys_last_update_block_height_idx ON public.access_keys_reindexed USING btree (last_update_block_height); DROP INDEX access_keys_public_key_idx; CREATE INDEX access_keys_public_key_idx ON public.access_keys_reindexed USING btree (public_key);

ALTER TABLE ONLY public.access_keys DROP CONSTRAINT created_by_receipt_fk; ALTER TABLE ONLY public.access_keys_reindexed ADD CONSTRAINT created_by_receipt_fk FOREIGN KEY (created_by_receipt_id) REFERENCES public.receipts(receipt_id) ON DELETE CASCADE; ALTER TABLE ONLY public.access_keys DROP CONSTRAINT deleted_by_receipt_fk; ALTER TABLE ONLY public.access_keys_reindexed ADD CONSTRAINT deleted_by_receipt_fk FOREIGN KEY (deleted_by_receipt_id) REFERENCES public.receipts(receipt_id) ON DELETE CASCADE;

ALTER TABLE access_keys RENAME TO access_keys_bk; ALTER TABLE access_keys_reindexed RENAME TO access_keys;

COMMIT;

7. Revert `indexer.service` and start (will use previous binary and start 500 blocks earlier than interruption)
```bash
systemctl start indexer
frol commented 1 year ago

@morgsmccauley The planned steps look good to me. The only nit is that you should change the --height 0 to match the block height of the genesis block (either check the genesis.json file or see the genesis height on Explorer for the respective network: https://explorer.near.org/stats).

I only wanted to chime in with the spare pair of eyes on the migration plan, and from here I am going to leave the code review and the rest of the operations to @khorolets.

morgsmccauley commented 1 year ago

Thanks @frol, updated --height 👍🏽

morgsmccauley commented 1 year ago

As we are in the process of migrating to the lake-based indexer, I'm going to wait until that is complete before reindexing this table. The bug fix for #304 still hasn't been deployed to production and doing so would be wasted work as they'll soon be shut down.

I'll raise another PR in the lake fork when ready to reindex. The plan will most likely remain the same but the patch may differ slightly. I'll also lump this work in with "storing genesis" as that's required for this reindex.