search

near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.31k stars 614 forks source link

`Corruption: block checksum mismatch` error #7515

Open zavodil opened 2 years ago

zavodil commented 2 years ago

Describe the bug Process was killed because of error:

Aug 08 15:52:56 near-4 neard[678063]: 2022-08-08T13:52:56.929051Z  INFO stats: #71226441 BbNdAgo1xwxnsoLBwzxHDP4XqMTJyMErj3cmy8iuJ6c8 100 validators 1 peer ⬇ 2.04 kB/s ⬆ 942 B/s 0.00 bps 0 gas/s CPU: 1118%, Mem: 65.2 GB
Aug 08 15:53:07 near-4 neard[678063]: 2022-08-08T13:53:07.069725Z ERROR network: Failed to remove expired peers err=Corruption: block checksum mismatch: stored = 4267500034, computed = 1650375527, type = 1  in /root/.near/data/6499719.sst offset 27326345 size 12631
Aug 08 15:53:31 near-4 neard[678063]: 2022-08-08T13:53:31.501053Z  INFO near_network::peer_manager::peer_manager_actor: Bandwidth stats total_bandwidth_used_by_all_peers=41659 total_msg_received_count=91 max_max_record_num_messages_in_progress=8
Aug 08 15:53:44 near-4 neard[678063]: 2022-08-08T13:53:44.198358Z  WARN chain: Failed to save processed height 71517930: IO Error: Corruption: block checksum mismatch: stored = 4267500034, computed = 1650375527, type = 1  in /root/.near/data/6499719.sst offset 27326345 size 12631
Aug 08 15:53:44 near-4 neard[678063]: 2022-08-08T13:53:44.202955Z ERROR near_client::client_actor: Error while committing largest skipped height IOErr(Custom { kind: Other, error: DBError("Corruption: block checksum mismatch: stored = 4267500034, computed = 1650375527, type = 1  in /root/.near/data/6499719.sst offset 27326345 size 12631") })
Aug 08 15:53:44 near-4 neard[678063]: 2022-08-08T13:53:44.212846Z  INFO stats: #71226441 BbNdAgo1xwxnsoLBwzxHDP4XqMTJyMErj3cmy8iuJ6c8 100 validators 1 peer ⬇ 2.04 kB/s ⬆ 942 B/s 0.00 bps 0 gas/s CPU: 990%, Mem: 65.3 GB
Aug 08 15:54:08 near-4 neard[678063]: 2022-08-08T13:54:08.058268Z ERROR network: Failed to remove expired peers err=Corruption: block checksum mismatch: stored = 4267500034, computed = 1650375527, type = 1  in /root/.near/data/6499719.sst offset 27326345 size 12631
Aug 08 15:54:31 near-4 neard[678063]: 2022-08-08T13:54:31.518854Z  INFO near_network::peer_manager::peer_manager_actor: Bandwidth stats total_bandwidth_used_by_all_peers=43650 total_msg_received_count=107 max_max_record_num_messages_in_progress=5
Aug 08 15:54:41 near-4 systemd[1]: neard.service: Main process exited, code=killed, status=9/KILL
Aug 08 15:54:41 near-4 systemd[1]: neard.service: Failed with result 'signal'.

Version (please complete the following information):

SamLegends commented 2 years ago

@marcin_near thought this might be related. The process wasn't killed, but I started missing blocks.

image

After trying to restart.

Sep 05 17:54:50 near-testnet-2 neard[248979]: 2022-09-05T17:54:50.334495Z ERROR network: Failed to remove expired peers err=Corruption: block checksum mismatch: stored = 2324967102, computed = 3394120547, type = 1 in /home/neard/.near/testnet/data/5059638.sst offset 43508645 size 13309 Sep 05 17:54:50 near-testnet-2 neard[248979]: 2022-09-05T17:54:50.917558Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 5%, Mem: 640 MB Sep 05 17:54:50 near-testnet-2 neard[248979]: 2022-09-05T17:54:50.917670Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:55:00 near-testnet-2 neard[248979]: 2022-09-05T17:55:00.918858Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 5%, Mem: 640 MB Sep 05 17:55:00 near-testnet-2 neard[248979]: 2022-09-05T17:55:00.918964Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:55:10 near-testnet-2 neard[248979]: 2022-09-05T17:55:10.919920Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 6%, Mem: 640 MB Sep 05 17:55:10 near-testnet-2 neard[248979]: 2022-09-05T17:55:10.920015Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:55:20 near-testnet-2 neard[248979]: 2022-09-05T17:55:20.920778Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 6%, Mem: 640 MB Sep 05 17:55:20 near-testnet-2 neard[248979]: 2022-09-05T17:55:20.920878Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:55:30 near-testnet-2 neard[248979]: 2022-09-05T17:55:30.922332Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 5%, Mem: 640 MB Sep 05 17:55:30 near-testnet-2 neard[248979]: 2022-09-05T17:55:30.922437Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:55:34 near-testnet-2 neard[248979]: 2022-09-05T17:55:34.854163Z INFO near_network::peer_manager::peer_manager_actor: Bandwidth stats total_bandwidth_used_by_all_peers=0 total_msg_received_count=0 max_max_record_num_messages_in_progress=0 Sep 05 17:55:40 near-testnet-2 neard[248979]: 2022-09-05T17:55:40.923144Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 5%, Mem: 640 MB Sep 05 17:55:40 near-testnet-2 neard[248979]: 2022-09-05T17:55:40.923236Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:55:50 near-testnet-2 neard[248979]: 2022-09-05T17:55:50.336253Z ERROR network: Failed to remove expired peers err=Corruption: block checksum mismatch: stored = 2324967102, computed = 3394120547, type = 1 in /home/neard/.near/testnet/data/5059638.sst offset 43508645 size 13309 Sep 05 17:55:50 near-testnet-2 neard[248979]: 2022-09-05T17:55:50.923723Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 5%, Mem: 641 MB Sep 05 17:55:50 near-testnet-2 neard[248979]: 2022-09-05T17:55:50.923834Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:55:56 near-testnet-2 neard[248979]: 2022-09-05T17:55:56.015665Z ERROR network: Failed to save peer data err=Corruption: block checksum mismatch: stored = 2324967102, computed = 3394120547, type = 1 in /home/neard/.near/testnet/data/5059638.sst offset 43508645 size 13309 Sep 05 17:56:00 near-testnet-2 neard[248979]: 2022-09-05T17:56:00.924499Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 7%, Mem: 640 MB Sep 05 17:56:00 near-testnet-2 neard[248979]: 2022-09-05T17:56:00.924598Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:56:10 near-testnet-2 neard[248979]: 2022-09-05T17:56:10.926388Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 6%, Mem: 639 MB Sep 05 17:56:10 near-testnet-2 neard[248979]: 2022-09-05T17:56:10.926495Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:56:20 near-testnet-2 neard[248979]: 2022-09-05T17:56:20.927963Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 5%, Mem: 639 MB Sep 05 17:56:20 near-testnet-2 neard[248979]: 2022-09-05T17:56:20.928065Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:56:30 near-testnet-2 neard[248979]: 2022-09-05T17:56:30.928584Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 5%, Mem: 639 MB Sep 05 17:56:30 near-testnet-2 neard[248979]: 2022-09-05T17:56:30.928678Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0 Sep 05 17:56:34 near-testnet-2 neard[248979]: 2022-09-05T17:56:34.856058Z INFO near_network::peer_manager::peer_manager_actor: Bandwidth stats total_bandwidth_used_by_all_peers=0 total_msg_received_count=0 max_max_record_num_messages_in_progress=0 Sep 05 17:56:40 near-testnet-2 neard[248979]: 2022-09-05T17:56:40.929554Z INFO stats: #99391507 Waiting for peers 0 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 5%, Mem: 639 MB Sep 05 17:56:40 near-testnet-2 neard[248979]: 2022-09-05T17:56:40.929651Z DEBUG stats: EpochId(EVhETg8X5q84pneGaWywQQsAaciprbyKqew4V8VFvfqs) Orphans: 0 With missing chunks: 0 In processing 0

pc-quiknode commented 3 months ago

Getting the same issue on our archive node. The node is stuck at block 119212622

Version

neard (release 1.39.2) (build 1.39.2) (rustc 1.75.0) (protocol 66) (db 38)

Error

ERROR near_client::client_actor: Error while committing largest skipped height IOErr(Custom { kind: Other, error: Error { message: "Corruption: block checksum mismatch: stored = 3417798032, computed = 25430532, type = 4  in /home/near/neard/data/60902278.sst offset 33291664 size 1282591" } })