Incomprehensible ForeignKeyViolation issue

[klefevre]

Hi everyone,

I'm trying to fully index the mainnet chain and I'm running into an issue that I just can't figure out.

---

First of all, I understood that I needed to import genesis data into my DB but I didn't find a clear explanation on how to get them. I finally found a very hackish way: I did checkout the branch master-nearcore and by following the README, I then, ran the commands:

cargo run --release -- --home-dir ~/.near/mainnet init --chain-id mainnet --download-config --download-genesis
cargo run --release -- --home-dir ~/.near/mainnet run --store-genesis --concurrency 1 sync-from-block --height 1

As the indexer starts by initializing genesis data before syncing with the network. Both tables accounts and access_keys were populated with respectively 26 and 35 rows.

First questions:

• Is this genesis data correct to you (as it seems quite small to me...)
• Is there an easier way to get them? (I've exported these rows to avoid having to do this again)

---

Then, I started to use the current version that use the Near Lake framework and I can't figure out why it stuck on the block #9823033 with the following error:

ERROR explorer_database:Error occurred during DatabaseError(ForeignKeyViolation, "insert or update on table \"account_changes\" violates foreign key constraint \"account_id_fk\""):
"AccountChanges were stored in database"
[
    AccountChange {
        affected_account_id: "erm.near",
        changed_in_block_timestamp: BigDecimal("1595370903490523743"),
        changed_in_block_hash: "4YM3tuxck9uHaqzr9cB5XSjup6C6oTWMsKL5YvQeoXhd",
        caused_by_transaction_hash: None,
        caused_by_receipt_id: Some(
            "EgNre6vjcPhMBZXCK6ApPYCxSnpCJA3VRqSFTtotz5BH",
        ),
        update_reason: ReceiptProcessing,
        affected_account_nonstaked_balance: BigDecimal("50000000000000000000000000"),
        affected_account_staked_balance: BigDecimal("0"),
        affected_account_storage_usage: BigDecimal("182"),
        index_in_block: 0,
    },
]

I use the cmd:

RUST_LOG=info cargo run --bin indexer-explorer -- --concurrency 1 mainnet from-block 1

Of course, when I use the option --non-strict-mode and optionally increase the concurrency. Everything seems to work perfectly but I merely see only the blocks table growing.

Questions:

• What can I do to index correctly this block? What am I missing?
• Should I trust data fetched using --non-strict-mode? For instance, I'm surprised that a table such as account_changes don't grow even after thousands of indexed blocks.
• In the event of a fix of my issue regarding this block. Would you recommend to use the --concurrency option e.g. --concurrency=100 (like the doc propose) in strict mode to speed up the process?

I know that's a lot of question but any help regarding all of this would be greatly appreciated.

Best,

[khorolets]

Thank you for bringing this up! We will need to investigate it.

Sorry for inconvenience, you did a great job to hack the store genesis we haven't ported to the new version yet (ref #327)

We will check the issue.

Meanwhile I can propose you to grab the missing piece from the Explorer DB using the public access mentioned in the README. You can't rely on the data collected with non-strict mode if you require a consistent data (non-strict mode skips data you might need)

cc @telezhnaya @morgsmccauley

[klefevre]

Hi, thanks for the follow up.

I wanted to give some news regarding my issue.

After an uncalculable number of retry (I use a container with --restart=unless-stopped in combination with the argument from-interruption) the block got finally fetched correctly. I don't really understand how and why it ended working as I absolutely did nothing more to make it work.

As of now, I haven't see any issues during the indexation. Sometime I see some errors but the internal retry does perfectly its job. @khorolets I assume this issue could be closed but I let you choose what to do if you want to keep investigate.

Best

[khorolets]

@klefevre thank you for the update. I would rather keep this issue open in order to check and improve that piece (if needed)