search

MinaProtocol / mina

Mina is a cryptocurrency protocol with a constant size blockchain, improving scaling while maintaining decentralization and security.
https://minaprotocol.com
Apache License 2.0
1.99k stars 529 forks source link

Mainnet migration - mina-berkeley-migration-script initial crashes with Attempt to register a file descriptor with Async that Async believes it is already managing #15550

Open jrwashburn opened 5 months ago

jrwashburn commented 5 months ago

Preliminary Checks

Description

Similar error to: https://github.com/MinaProtocol/mina/issues/14414

Running initial mainnet migration with this command:

mina-berkeley-migration-script initial --genesis-ledger /var/lib/coda/mainnet.json --source-db "postgresql://*/mainnet-archive" --target-db "postgresql://*/mainnet-archive-migrated" --blocks-bucket mina-mainnet-blocks --blocks-batch-size 2000 --checkpoint-output-path . --precomputed-blocks-local-path /mnt/volume_nyc3_01/migration/ --network mainnet >~/migration/initial-migration-attempt5.log 2>&1

Fails after a while (first attempt got through 200k blocks, second attempt (attempt 5 log) got through 120k blocks. Also causes postgres failure so I suspect this is related to file limits.

My ulimit -n was 1024, updated to 4096 and trying again.

Steps to Reproduce

  1. Start initial migration

  3. ...

Expected Result

Migration will complete

Actual Result

Migration fails with error attached and also crashes postgresql. initial-migration-attempt5.log

Daemon version

mina-archive-migration/unknown,now 3.0.1upgrade-tooling amd64 [installed]

How frequently do you see this issue?

Always

What is the impact of this issue on your ability to run a node?

Low

Status

not relevant

Additional information

No response

kaozenn commented 5 months ago

This issue isn't blocking, and rerunning the job helps to move forward. Unless there is a fix for the root cause, I am suggesting to add a retry mechanism to help mitigate this issue.

dkijania commented 5 months ago

@jrwashburn can you decrease --block-batch-size to values around 500 ? It shouldn't impact migration speed greatly and it is safer to run migration with smaller batch size args

jrwashburn commented 5 months ago

I have retried enough to be down to the last 1112 blocks, so I cannot reproduce now. I tried with batch size 100, but failing with a different error, which I created a new issue for: https://github.com/MinaProtocol/mina/issues/15554