Help with importing blocks

[garethtdavies]

Using mina-indexer --version mina-indexer 0.1.1

Trying to understand the process for importing blocks. As I understand the precomputed blocks have to be in a local directory, and that directory is passed via the --logs-dir flag. Is the naming/format of the blocks as is currently in Gcloud?

If so, can I import an arbitrary block for testing purposes, e.g., this block https://storage.googleapis.com/mina_network_block_data/mainnet-100001-3NLMfBr61BrTrZ86BKsJBUtWvRaGEyCBYTwywL5Ut7Jqye2qkfvv.json which is stored in the /home/gareth/logs/ directory.

I don't get any meaningful output when doing this, so I assume I am missing something.

=== Run ===
Starting mina-indexer with logs dir "/home/gareth/logs/"

[Isaac-DeFrain]

Hey @garethtdavies, thanks for taking an interest in our project!

Unfortunately, that CLI is not currently in a working state. Most commands just print a little message as you've discovered. There is a way to do something similar to what you want, maybe @jenr24G can say more about it. Our docs need some work to reflect the current state of the project, thanks for pointing our attention to that!

The goal is to be able to do exactly what you're trying to do, given a starting ledger in addition to the starting block. We're also working on a snapshotting mechanism to quickly sync the db and internal state. Stay tuned for that!

Is the naming/format of the blocks as is currently in Gcloud? Yes.

Also, if interested, see PR #13258 which introduces local block dumping in that same format to the Mina daemon.

[jenr24G]

Hi @garethtdavies ! I'm glad you're taking an interest in our project!

Currently, the mina-indexer-server binary can import precomputed block logs following the GCloud format, although they have to be preprocessed first due to a bug in serde_json. Effectively what needs to happen is that the blocks coming from the daemon need to have jq run on them before parsing to introduce newlines into the data. We're working on raising an issue with the maintainers of serde_json, and in the meantime, I'm working on a branch that uses jq_rs to do the preprocessing automatically, at origin/jq-parsing.

If you don't want to pull down my in progress branch, you could simply run for block in mainnet/*.json; do jq $block > "${block}_tmp" && mv "${block}_tmp" $block done then run mina-indexer-server on the preprocessed blocks. I've confirmed this method works on a local mina network, using the scripts in scripts/mina-local-network/ from the Mina repository

We're planning on having a more robust solution for this by the end of the month, along with a GraphQL interface using (a subset of) the MinaExplorer GQL schema you provided to me, that should work on the mainnet blocks!

Best of luck, and I hope you like what we're working on!

Jen

[Isaac-DeFrain]

Hey @garethtdavies! We have at last resolved the above issues among others 🎉

Very soon we will add restoration from a rocksdb snapshot and the ability to initialize from an arbitrary (canonical) ledger among other things! However, for the moment, you can only initialize from the mainnet genesis ledger, using any collection of mainnet blocks ranging from all canonical blocks to all blocks.

After entering the nix shell, just do:

cargo run --release --bin mina-indexer -- server -i -l path/to/mainnet.json -s path/to/mainnet/blocks/dir

and you'll be indexing like a pro! 😄

We're open to and appreciate any and all feedback!

Happy indexing!

[robinbb]

To show completion of this, a passing CI test is required. Re-opening.

[trevorbernard]

Resolved