search

AcalaNetwork / Acala

Acala - cross-chain DeFi hub and stablecoin based on Substrate for Polkadot and Kusama.
https://acala.network
GNU General Public License v3.0
741 stars 456 forks source link

Acala mainnet stuck on finalized block #5446885 #2774

Closed andreclaro closed 1 month ago

andreclaro commented 2 months ago

Describe the bug A new acala mainnet node, which has been syncing from genesis, is stuck on finalized block #5446885

Expected Behavior Sync from genesis till the latest block without issues.

Current Behavior Suck on finalized block #5446885.

Steps to Reproduce

  1. Start acala client without database
  2. Wait a few days

Additional context

andreclaro commented 2 months ago

last days - block height image

last 12 hours - block height image

andreclaro commented 2 months ago

I see a few message like Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.

Example:

Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -1024 to -2147483648. Reason: same small block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
eskimor commented 2 months ago

Does a restart help?

andreclaro commented 2 months ago

No, it didn't help. I tried multiple times to restart the node.

Now, I deleted the acala database and the node is syncing from genesis again... I will update this issue...

image

eskimor commented 2 months ago

Do you still have the database around? Restarting with debug or even trace logs might reveal what is going on. 

RUST_LOG=debug /your/collator/binary --args-you-normally-pass

Trace for everything is likely to spammy, but only for sync should work:

RUST_LOG=sync=trace /your/collator/binary --args-you-normally-pass
andreclaro commented 2 months ago

I will enable that level of logging if I see this or any other issues. Thanks

andreclaro commented 1 month ago

I didn't see this issue again. Closing for now.