mina-berkeley-migration-script improve error message for missing blocks failure

jrwashburn commented 6 months ago

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
[X] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions

Description

mina-berkeley-migration-script initial fails with

(monitor.ml.Error
 (Not_found_s
  ("Map.find_exn: not found"
   2073082141344616072397527199815941276656439868491454361417667551078491250181))
 ("Raised at Base__Error.raise in file \"src/error.ml\" (inlined), line 8, characters 14-30"
  "Called from Caqti_async.Fiber.finally.(fun) in file \"caqti-async/lib/caqti_async.ml\", line 44, characters 39-69"
  "Called from Async_kernel__Deferred1.M.map.(fun) in file \"src/deferred1.ml\", line 17, characters 40-45"
  "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
  "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"))

Steps to Reproduce

Run mina-berkeley-migration-script initial (full command w/ params below)
...

Expected Result

Migration will complete

Actual Result

Migration fails with error.

Command:

mina-berkeley-migration-script initial --genesis-ledger /var/lib/coda/mainnet.json --source-db "postgresql://***/mainnet-archive" --target-db "postgresql://***/mainnet-archive-migrated" --blocks-bucket mina-mainnet-blocks --blocks-batch-size 2000 --checkpoint-output-path . --precomputed-blocks-local-path /mnt/volume_nyc3_01/migration/ --network mainnet >~/migration/initial-migration-attempt8.log 2>&1

Output:

2024-04-21 15:53:24 UTC [Info] Loaded genesis ledger from $ledger_file
        ledger_file: "/tmp/coda_cache_dir/genesis_ledger_accounts_fe217061fe891753eec9838e1f4edc169ff6c7f28ad0e9628f2167e6716f5fec.tar.gz"
2024-04-21 15:53:24 UTC [Info] Got precomputed values from runtime config
2024-04-21 15:53:25 UTC [Info] Querying mainnet canonical blocks
2024-04-21 15:53:27 UTC [Info] Determining already migrated bocks
2024-04-21 15:53:27 UTC [Info] Already migrated blocks through height 345000, resuming migration
2024-04-21 15:53:27 UTC [Info] Will migrate 1112 mainnet blocks
2024-04-21 15:53:27 UTC [Info] Migrating mainnet blocks
2024-04-21 15:53:27 UTC [Info] Migrating 1112 blocks starting at height 345001 (3NKdMg63gzE9MGQs3VaWDpdiBwagGMjpSeiRDYMHWckw5QpDytBJ..3NLSKLgtTHmbbJYapW6zW4a3to1kmWiQANQX2Sagph4yRt2ytayr)
2024-04-21 15:53:27 UTC [Info] Fetching batch of precomputed blocks
2024-04-21 15:53:38 UTC [Info] Found 346111 individually downloaded precomputed blocks
2024-04-21 15:53:38 UTC [Info] Will download 0 precomputed blocks (this may take a while)
2024-04-21 15:54:02 UTC [Info] Converting blocks to extensional format...
2024-04-21 15:54:02 UTC [Info] Fetching transaction sequence from prior database
2024-04-21 15:54:04 UTC [Info] Done fetching transaction sequence from prior database
2024-04-21 15:54:04 UTC [Info] Adding blocks to migrated database...
(monitor.ml.Error
 (Not_found_s
  ("Map.find_exn: not found"
   2073082141344616072397527199815941276656439868491454361417667551078491250181))
 ("Raised at Base__Error.raise in file \"src/error.ml\" (inlined), line 8, characters 14-30"
  "Called from Caqti_async.Fiber.finally.(fun) in file \"caqti-async/lib/caqti_async.ml\", line 44, characters 39-69"
  "Called from Async_kernel__Deferred1.M.map.(fun) in file \"src/deferred1.ml\", line 17, characters 40-45"
  "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
  "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"))

Daemon version

mina-archive-migration/unknown,now 3.0.1upgrade-tooling amd64 [installed]

How frequently do you see this issue?

Always

What is the impact of this issue on your ability to run a node?

Blocker

Status

mina client status is not relevant

Additional information

I got the same error trying with --blocks-batch-size 100.

dkijania commented 6 months ago

@jrwashburn can you check if your source database has any missing blocks ? Usually this issue means that there is a gap in chain. If that is true let's keep it open as we need to improve error reporting

jrwashburn commented 6 months ago

There are missing blocks - I forgot I stopped the node while running migration but then started again so there is a gap. I'll fill it and retry.

jrwashburn commented 6 months ago

Sorry - user error; filling the gaps resolved. Closing.

jrwashburn commented 6 months ago

Oops - reopening to improve error message.

MinaProtocol / mina