search

bnb-chain / node-binary

Binaries for full nodes, light-weighted clients
178 stars 116 forks source link

Node failing to synchronize using snapshot #253

Open Eric-Showers opened 4 years ago

Eric-Showers commented 4 years ago

Steps followed:

1) Installed node on Ubuntu 18.04.3 using this article 2) Downloaded most recent snapshot from this article and followed instructions 3) Modified config.toml

4) Ran bnbchaind start, which produced this error:

panic: Failed to process committed block (112397623:6EA3AE907079229A36C4F8F12E00FB682BEF23A6349419722C76FA14F13EC329): Wrong Block.Header.AppHash. Expected F9F03780A35803BFE10F60A5814D48ED059016032CD04075D44BA3233B559D98, got ACE158FBC230B0897622D8D8913FC538F2B73A51B9C9627C4A66D5F081AD7C84

goroutine 58419 [running]: github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc00b0469c0) /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.3/blockchain/v0/reactor.go:404 +0x1c70 created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.3/blockchain/v0/reactor.go:127 +0x84

5) Attempted running again, same error:

ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F9F03780A35803BFE10F60A5814D48ED059016032CD04075D44BA3233B559D98, got ACE158FBC230B0897622D8D8913FC538F2B73A51B9C9627C4A66D5F081AD7C84

Was it a mistake to turn off state sync? I need this node to have access to all historical blocks so I assumed that fast sync would be required.

chainwhisper commented 4 years ago

Hello！

Which version of bnbchaiind did you use?

Eric-Showers commented 4 years ago 
bnbchaind version
Binance Chain Release: v0.8.0;Binance Chain Commit: 422f0a96; Cosmos Release: v0.25.0-binance.23; Tendermint Release: v0.32.3-binance.3;
chainwhisper commented 4 years ago

@Eric-Showers Have you verify if app.toml file is the latest?

Eric-Showers commented 4 years ago

Yes sir - my app.toml file is identical to the one found here

chainwhisper commented 4 years ago

@Eric-Showers, please verify the sha256sum: 194cbad7e1adcaafc16df94576f0a3b4a26245e4019cef93c51a3183bef71d73

Eric-Showers commented 4 years ago

Okay I just completed a sha256 hash of the compressed snapshot and it matches that checksum

alexqrid commented 3 years ago

I confirm that the issue still presents. I downloaded Latest snapshot of February 1st and tried to sync from it on linux server. I used the latest version of the binary for linux - 0.8.2.

:~/Binance/node-binary/fullnode/prod/0.8.2/linux# uname -a
Linux server 5.11.0-051100-generic #202102142330 SMP Sun Feb 14 23:33:21 UTC 2021 x86_64 GNU/Linux

But the node crashed.

Logs ```sh I[2021-03-01|12:41:44.161] replayed all tx. Starting match module=dex height=142231523 I[2021-03-01|12:41:44.161] No order comes in for the block module=dexkeeper I[2021-03-01|12:41:44.161] Relaying block for order book module=dex height=142231524 I[2021-03-01|12:41:44.161] Canceled Order module=dex order="CancelOrderMsg{Sender:FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B, RefId: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B-960881}" I[2021-03-01|12:41:44.161] Canceled Order module=dex order="CancelOrderMsg{Sender:DF6136BC4D4E9756437DBB59F45EC0468B2DDEB1, RefId: DF6136BC4D4E9756437DBB59F45EC0468B2DDEB1-550255}" I[2021-03-01|12:41:44.161] Canceled Order module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211025}" I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2210936}" I[2021-03-01|12:41:44.162] Added Order module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211035, Symbol: FRM-DE7_BNB}" I[2021-03-01|12:41:44.162] replayed all tx. Starting match module=dex height=142231524 I[2021-03-01|12:41:44.162] Relaying block for order book module=dex height=142231525 I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:07285195B9434C05899780381C0E7372C7A03591, RefId: 07285195B9434C05899780381C0E7372C7A03591-340023}" I[2021-03-01|12:41:44.162] Added Order module=dex order="NewOrderMsg{Sender: 612F1135E2A7A514C4444C526297153E1007E16E, Id: 612F1135E2A7A514C4444C526297153E1007E16E-3662609, Symbol: NEXO-A84_BNB}" I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211018}" I[2021-03-01|12:41:44.162] Added Order module=dex order="NewOrderMsg{Sender: D089834ACD87C38C46D00E3BCE7160A16B69F002, Id: D089834ACD87C38C46D00E3BCE7160A16B69F002-346350, Symbol: XRP-BF2_BNB}" I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:7CD235E3C90C2D389E295CC6B4E4BDF445671D44, RefId: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44-653784}" I[2021-03-01|12:41:44.162] Added Order module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874960, Symbol: BULL-BE4_BNB}" I[2021-03-01|12:41:44.162] replayed all tx. Starting match module=dex height=142231525 I[2021-03-01|12:41:44.162] Relaying block for order book module=dex height=142231526 I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2210981}" I[2021-03-01|12:41:44.162] replayed all tx. Starting match module=dex height=142231526 I[2021-03-01|12:41:44.162] No order comes in for the block module=dexkeeper I[2021-03-01|12:41:44.162] Relaying block for order book module=dex height=142231527 I[2021-03-01|12:41:44.162] Added Order module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211038, Symbol: TWT-8C2_BNB}" I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:5CC9EEA33C86D854E0A3F8353111BC88C3FD0DAF, RefId: 5CC9EEA33C86D854E0A3F8353111BC88C3FD0DAF-73508}" I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:B8606C02542B4B41BFDDE81B9AF7776BF73486BB, RefId: B8606C02542B4B41BFDDE81B9AF7776BF73486BB-890893}" I[2021-03-01|12:41:44.162] replayed all tx. Starting match module=dex height=142231527 I[2021-03-01|12:41:44.162] Relaying block for order book module=dex height=142231528 I[2021-03-01|12:41:44.162] replayed all tx. Starting match module=dex height=142231528 I[2021-03-01|12:41:44.162] No order comes in for the block module=dexkeeper I[2021-03-01|12:41:44.162] Relaying block for order book module=dex height=142231529 I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:612F1135E2A7A514C4444C526297153E1007E16E, RefId: 612F1135 E2A7A514C4444C526297153E1007E16E-3662552}" I[2021-03-01|12:41:44.162] Added Order module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198D E83FDDE4BC0BDF3F3A5621CF6A8-2211039, Symbol: SHR-DB6_BNB}" I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF5 9198DE83FDDE4BC0BDF3F3A5621CF6A8-2210982}" I[2021-03-01|12:41:44.162] Canceled Order module=dex order="CancelOrderMsg{Sender:2C3ACBDA0A3DA42FCE4AC4D6750BC371C0BFC176, RefId: 2C3ACBDA 0A3DA42FCE4AC4D6750BC371C0BFC176-710093}" I[2021-03-01|12:41:44.162] Added Order module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3AD C9CFF3EF31ADC7EDD288F5E208E-27874961, Symbol: MITH-C76_BNB}" I[2021-03-01|12:41:44.162] Added Order module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198D E83FDDE4BC0BDF3F3A5621CF6A8-2211041, Symbol: NOW-E68_BNB}" I[2021-03-01|12:41:44.162] replayed all tx. Starting match module=dex height=142231529 I[2021-03-01|12:41:44.163] Relaying block for order book module=dex height=142231530 I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198D E83FDDE4BC0BDF3F3A5621CF6A8-2211042, Symbol: TWT-8C2_BNB}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: DF6136BC4D4E9756437DBB59F45EC0468B2DDEB1, Id: DF6136BC4D4E9 756437DBB59F45EC0468B2DDEB1-550263, Symbol: MITX-CAA_BNB}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:1EB0FCA540E10FE79867804B3BF5034BFC9D2A12, RefId: 1EB0FCA540E10FE79867804B3BF5034BFC9D2A12-2134187}" I[2021-03-01|12:41:44.163] replayed all tx. Starting match module=dex height=142231530 I[2021-03-01|12:41:44.163] Relaying block for order book module=dex height=142231531 I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874962, Symbol: BULL-BE4_BNB}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2210987}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211044, Symbol: CBIX-3C9_BNB}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: B9E7F47CB15FA05A6900818F142B09213C265758, Id: B9E7F47CB15FA05A6900818F142B09213C265758-750644, Symbol: FRM-DE7_BNB}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 612F1135E2A7A514C4444C526297153E1007E16E, Id: 612F1135E2A7A514C4444C526297153E1007E16E-3662611, Symbol: CAN-677_BNB}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874963, Symbol: ETHBULL-D33_BNB}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A, RefId: 4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A-319059}" I[2021-03-01|12:41:44.163] replayed all tx. Starting match module=dex height=142231531 I[2021-03-01|12:41:44.163] Relaying block for order book module=dex height=142231532 I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:5AEBC595B8C8EDBFF616D9CBF291BE12E46C9AE8, RefId: 5AEBC595B8C8EDBFF616D9CBF291BE12E46C9AE8-620018}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A, Id: 4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A-319080, Symbol: CHZ-ECD_BNB}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B, Id: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B-960883, Symbol: NEXO-A84_BNB}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:AB3283B32936A6F2BBD7942F93716D96B1EE3A0E, RefId: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E-1573554}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:BBE4CDB5EFC7AF0E92EA467A03169370E6249BE8, RefId: BBE4CDB5EFC7AF0E92EA467A03169370E6249BE8-360780}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:7CD235E3C90C2D389E295CC6B4E4BDF445671D44, RefId: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44-653795}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E, Id: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E-1573556, Symbol: BTTB-D31_BNB}" I[2021-03-01|12:41:44.163] replayed all tx. Starting match module=dex height=142231532 I[2021-03-01|12:41:44.163] Relaying block for order book module=dex height=142231533 I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:8452761BA0E6E214AF58CA493294263B81AE5A32, RefId: 8452761BA0E6E214AF58CA493294263B81AE5A32-6482184}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 8452761BA0E6E214AF58CA493294263B81AE5A32, Id: 8452761BA0E6E214AF58CA493294263B81AE5A32-6482206, Symbol: DOS-120_BNB}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, RefId: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874952}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44, Id: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44-653800, Symbol: XRP-BF2_BUSD-BD1}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: ABF1725319F1ACFC83382D2D084D75B97EBEC306, Id: ABF1725319F1ACFC83382D2D084D75B97EBEC306-625052, Symbol: XRP-BF2_BUSD-BD1}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, RefId: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874962}" I[2021-03-01|12:41:44.164] Canceled Order module=dex order="CancelOrderMsg{Sender:0F674A63B7B3E956CF0A206A6195AF3090C0649E, RefId: 0F674A63B7B3E956CF0A206A6195AF3090C0649E-467060}" I[2021-03-01|12:41:44.164] Added Order module=dex order="NewOrderMsg{Sender: B8606C02542B4B41BFDDE81B9AF7776BF73486BB, Id: B8606C02542B4B41BFDDE81B9AF7776BF73486BB-890895, Symbol: CAS-167_BNB}" I[2021-03-01|12:41:44.164] Canceled Order module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2210991}" I[2021-03-01|12:41:44.164] replayed all tx. Starting match module=dex height=142231533 I[2021-03-01|12:41:44.164] Relaying block for order book module=dex height=142231534 I[2021-03-01|12:41:44.164] Canceled Order module=dex order="CancelOrderMsg{Sender:BC6E0EC1C60C6FBBC1AFB0A51F15013C9988E756, RefId: BC6E0EC1C60C6FBBC1AFB0A51F15013C9988E756-506353}" I[2021-03-01|12:41:44.164] Added Order module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198D E83FDDE4BC0BDF3F3A5621CF6A8-2211044, Symbol: CBIX-3C9_BNB}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: B9E7F47CB15FA05A6900818F142B09213C265758, Id: B9E7F47CB15FA 05A6900818F142B09213C265758-750644, Symbol: FRM-DE7_BNB}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 612F1135E2A7A514C4444C526297153E1007E16E, Id: 612F1135E2A7A 514C4444C526297153E1007E16E-3662611, Symbol: CAN-677_BNB}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3AD C9CFF3EF31ADC7EDD288F5E208E-27874963, Symbol: ETHBULL-D33_BNB}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A, RefId: 4BD42EEC CE4ADCF3EEC8FA67598F83DD6CACD36A-319059}" I[2021-03-01|12:41:44.163] replayed all tx. Starting match module=dex height=142231531 I[2021-03-01|12:41:44.163] Relaying block for order book module=dex height=142231532 I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:5AEBC595B8C8EDBFF616D9CBF291BE12E46C9AE8, RefId: 5AEBC595 B8C8EDBFF616D9CBF291BE12E46C9AE8-620018}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A, Id: 4BD42EECCE4AD CF3EEC8FA67598F83DD6CACD36A-319080, Symbol: CHZ-ECD_BNB}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B, Id: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B-960883, Symbol: NEXO-A84_BNB}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:AB3283B32936A6F2BBD7942F93716D96B1EE3A0E, RefId: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E-1573554}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:BBE4CDB5EFC7AF0E92EA467A03169370E6249BE8, RefId: BBE4CDB5EFC7AF0E92EA467A03169370E6249BE8-360780}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:7CD235E3C90C2D389E295CC6B4E4BDF445671D44, RefId: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44-653795}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E, Id: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E-1573556, Symbol: BTTB-D31_BNB}" I[2021-03-01|12:41:44.163] replayed all tx. Starting match module=dex height=142231532 I[2021-03-01|12:41:44.163] Relaying block for order book module=dex height=142231533 I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:8452761BA0E6E214AF58CA493294263B81AE5A32, RefId: 8452761BA0E6E214AF58CA493294263B81AE5A32-6482184}" I[2021-03-01|12:41:44.163] Added Order module=dex order="NewOrderMsg{Sender: 8452761BA0E6E214AF58CA493294263B81AE5A32, Id: 8452761BA0E6E214AF58CA493294263B81AE5A32-6482206, Symbol: DOS-120_BNB}" I[2021-03-01|12:41:44.163] Canceled Order module=dex order="CancelOrderMsg{Sender:1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, RefId: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874952}" .... I[2021-03-01|12:41:44.179] Added Order module=dex order="NewOrderMsg{Sender: 5CC9EEA33C86D854E0A3F8353111BC88C3FD0DAF, Id: 5CC9EEA33C86D 854E0A3F8353111BC88C3FD0DAF-73516, Symbol: DOS-120_BNB}" I[2021-03-01|12:41:44.179] replayed all tx. Starting match module=dex height=142231612 I[2021-03-01|12:41:46.255] Located day to load breathe block height module=dexkeeper epochDay="\ufffdH\u0000\u0000\u0000\u0000\u0000\u0000" I[2021-03-01|12:41:46.255] Loaded breathe block height module=dexkeeper height=142180985 I[2021-03-01|12:41:46.255] get last breathe block height module=main height=142180985 I[2021-03-01|12:41:46.255] reload latest snapshot module=statesync height=142180985 I[2021-03-01|12:41:48.457] ABCI Handshake App Info module=consensus height=142231612 hash=6AC28FEA5CC0BDD1D7F84D6A705F2C2243A318C612B8DE0EF0E73F63B9 586F24 software-version= protocol-version=0 I[2021-03-01|12:41:48.457] ABCI Replay Blocks module=consensus appHeight=142231612 storeHeight=142231613 stateHeight=142231612 I[2021-03-01|12:41:48.457] Replay last block using real app module=consensus I[2021-03-01|12:41:48.463] symbols to match module=dexkeeper symbols="[BEAR-14C_BNB MATIC-84A_BNB]" I[2021-03-01|12:41:48.472] Sync params proposals. module=paramHub I[2021-03-01|12:41:48.498] start to collect publish information module=pub height=142231613 I[2021-03-01|12:41:48.515] start to publish module=pub height=142231613 blockTime=1612418258924370991 numOfTrades=0 numOfOrders=4 numOfProposals=0 numOfSideProposals=0 numOfStakeUpdates=0 numOfAccounts=5 I[2021-03-01|12:41:48.515] Executed block module=consensus height=142231613 validTxs=4 invalidTxs=0 I[2021-03-01|12:41:48.540] Committed state module=consensus height=142231613 txs=4 appHash=775B775DF3348FA7242FF9735E07E64C5E53486375184F8E16EA4525B6CA4DDE I[2021-03-01|12:41:48.541] Completed ABCI Handshake - Tendermint and App are synced module=consensus appHeight=142231612 appHash=6AC28FEA5CC0BDD1D7F84D6A705F2C2243A318C612B8DE0EF0E73F63B9586F24 I[2021-03-01|12:41:48.541] This node is not a validator module=consensus addr=FB4560C8F3982AA272BF10FEAE472024D2397C78 pubKey=PubKeyEd25519{A94507950684BB9352761BDB09148CA7232C699B8CB3ADD94F2C76967FA45452} I[2021-03-01|12:41:48.563] Starting BlockchainReactor module=blockchain impl=BlockchainReactor I[2021-03-01|12:41:48.563] Starting Consensus module=consensus impl=ConsensusReactor I[2021-03-01|12:41:48.563] ConsensusReactor module=consensus fastSync=true I[2021-03-01|12:41:48.563] Starting StateSyncReactor module=statesync impl=StateSyncReactor I[2021-03-01|12:41:48.563] Starting StatePool module=statesync impl=StatePool I[2021-03-01|12:41:48.563] broadcast state status request module=statesync E[2021-03-01|12:41:48.658] failed to load immutable tree module=statesync err="version does not exist" store="KVStoreKey{0xc000218020, acc}" I[2021-03-01|12:41:48.752] added a peer to state sync reactor module=statesync peer=dbec1c96777dc757db67945cbc41e3e793b94261 E[2021-03-01|12:41:48.825] dialing failed (attempts: 1): auth failure: conn.ID (af89cf5aa3aff6d8b3f60043b8706dc458293b7f) dialed ID (aeed2359fde99ee784e0407e8e062be4b555a360) mismatch module=pex addr=aeed2359fde99ee784e0407e8e062be4b555a360@3.82.70.135:27146 E[2021-03-01|12:41:48.833] connection with 0fd066de431d13e5613aba53b3ed25c5a7a0b67c@13.114.123.99:27146 has been established or dialed module=pex addr=0fd066de431d13e5613aba53b3ed25c5a7a0b67c@13.114.123.99:27146 E[2021-03-01|12:41:48.833] connection with 7d2a3c3727f46a6d04321656ecdf2f75d1d9c0dc@52.54.234.10:27146 has been established or dialed module=pex addr=7d2a3c3727f46a6d04321656ecdf2f75d1d9c0dc@52.54.234.10:27146 E[2021-03-01|12:41:48.833] connection with bfa682d7d5bb503b724f8dc34a0ef5f6bfa17d6f@35.159.10.214:27146 has been established or dialed module=pex addr=bfa682d7d5bb503b724f8dc34a0ef5f6bfa17d6f@35.159.10.214:27146 E[2021-03-01|12:41:48.833] connection with 7156d461742e2a1e569fd68426009c4194830c93@52.193.47.207:27146 has been established or dialed module=pex addr=7156d461742e2a1e569fd68426009c4194830c93@52.193.47.207:27146 E[2021-03-01|12:41:48.833] connection with 1ebb7d9bc3ee3001f41200ed4f4be5432586c43e@34.92.4.164:27147 has been established or dialed module=pex addr=1ebb7d9bc3ee3001f41200ed4f4be5432586c43e@34.92.4.164:27147 E[2021-03-01|12:41:48.833] connection with 782303c9060d46211225662fdd1dd411c638263a@52.197.243.252:27146 has been established or dialed module=pex addr=782303c9060d46211225662fdd1dd411c638263a@52.197.243.252:27146 I[2021-03-01|12:41:48.946] added a peer to state sync reactor module=statesync peer=ca1245471ff1f2224e1076dfd94d7d2d66077cf5 I[2021-03-01|12:41:48.947] added a peer to state sync reactor module=statesync peer=7d2a3c3727f46a6d04321656ecdf2f75d1d9c0dc E[2021-03-01|12:41:48.953] dialing failed (attempts: 1): dial tcp 3.92.86.128:27146: connect: connection refused module=pex addr=8825b32e3abec71d772abf009ba1956d452be1fa@3.92.86.128:27146 E[2021-03-01|12:41:48.963] dialing failed (attempts: 1): dial tcp 35.208.230.254:27146: connect: connection refused module=pex addr=8e3e9bcc2802517c7cdec06748252307811242e3@35.208.230.254:27146 I[2021-03-01|12:41:49.049] added a peer to state sync reactor module=statesync peer=48dada912ff10f04dec0db7900a78750384d06fd I[2021-03-01|12:41:49.074] added a peer to state sync reactor module=statesync peer=e2c323d27867bad2298a5452490157436e70e698 I[2021-03-01|12:41:49.081] added a peer to state sync reactor module=statesync peer=cfb5c1f7c10dfd99219158f5491ce65227bfd072 E[2021-03-01|12:41:49.094] dialing failed (attempts: 1): auth failure: secret conn failed: EOF module=pex addr=381934b9b3f862d1e81e699d8e1d38929f330cef@54.158.222.212:27146 I[2021-03-01|12:41:49.097] added a peer to state sync reactor module=statesync peer=b323708b0bd429a95154ff0a287f0f42f432cb62 E[2021-03-01|12:41:49.130] connection with 29792743565c4470480dbf7ef43b9256465dfa74@99.81.122.88:27146 has been established or dialed module=pex addr=29792743565c4470480dbf7ef43b9256465dfa74@99.81.122.88:27146 E[2021-03-01|12:41:49.130] connection with 8c379d4d3b9995c712665dc9a9414dbde5b30483@54.249.16.115:27146 has been established or dialed module=pex addr=8c379d4d3b9995c712665dc9a9414dbde5b30483@54.249.16.115:27146 E[2021-03-01|12:41:49.130] connection with 7d2a3c3727f46a6d04321656ecdf2f75d1d9c0dc@52.54.234.10:27146 has been established or dialed module=pex addr=7d2a3c3727f46a6d04321656ecdf2f75d1d9c0dc@52.54.234.10:27146 E[2021-03-01|12:41:49.130] connection with 782303c9060d46211225662fdd1dd411c638263a@52.197.243.252:27146 has been established or dialed module=pex addr=782303c9060d46211225662fdd1dd411c638263a@52.197.243.252:27146 E[2021-03-01|12:41:49.130] connection with e2c323d27867bad2298a5452490157436e70e698@52.213.118.150:27146 has been established or dialed module=pex addr=e2c323d27867bad2298a5452490157436e70e698@52.213.118.150:27146 E[2021-03-01|12:41:49.130] connection with b7affdf084115fd7e7cdbc04686ef8c218e1801c@52.199.222.255:27146 has been established or dialed module=pex addr=b7affdf084115fd7e7cdbc04686ef8c218e1801c@52.199.222.255:27146 E[2021-03-01|12:41:49.130] connection with 229ec3d0e55be99dee2194142d0ffc33e923a30d@3.209.208.253:27146 has been established or dialed module=pex addr=229ec3d0e55be99dee2194142d0ffc33e923a30d@3.209.208.253:27146 E[2021-03-01|12:41:49.168] connection with 7d2a3c3727f46a6d04321656ecdf2f75d1d9c0dc@52.54.234.10:27146 has been established or dialed module=pex addr=7d2a3c3727f46a6d043 21656ecdf2f75d1d9c0dc@52.54.234.10:27146 E[2021-03-01|12:41:49.168] Stopping peer for error module=p2p peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 out}" err=EOF I[2021-03-01|12:41:49.170] added a peer to state sync reactor module=statesync peer=0fd066de431d13e5613aba53b3ed25c5a7a0b67c I[2021-03-01|12:41:49.174] added a peer to state sync reactor module=statesync peer=782303c9060d46211225662fdd1dd411c638263a I[2021-03-01|12:41:49.177] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 out}" I[2021-03-01|12:41:49.177] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 out}" I[2021-03-01|12:41:49.203] added a peer to state sync reactor module=statesync peer=289f680c0c78465dbf1d86c2260dd032d4541a4c I[2021-03-01|12:41:49.205] added a peer to state sync reactor module=statesync peer=229ec3d0e55be99dee2194142d0ffc33e923a30d I[2021-03-01|12:41:49.208] added a peer to state sync reactor module=statesync peer=e2acf608686f830386e8648d6fc2f50c3316aa88 E[2021-03-01|12:41:49.235] dialing failed (attempts: 1): incompatible: Peer is on a different network. Got Binance-Chain-Ganges, expected Binance-Chain-Tigris module=pex addr=7156d461742e2a1e569fd68426009c4194830c93@52.193.47.207:27146 E[2021-03-01|12:41:49.249] dialing failed (attempts: 1): dial tcp 52.200.132.60:27146: connect: connection refused module=pex addr=8825b32e3abec71d772abf009ba1956d452be1fa@52.200.132.60:27146 E[2021-03-01|12:41:49.256] dialing failed (attempts: 1): auth failure: conn.ID (98579505572cc85488caa403bebb62e5b61d68e0) dialed ID (bb721fe089e83d42bcb8f629f8efe0e4af888fde) mismatch module=pex addr=bb721fe089e83d42bcb8f629f8efe0e4af888fde@52.197.88.253:27146 E[2021-03-01|12:41:49.332] Stopping peer for error module=p2p peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}" err=EOF I[2021-03-01|12:41:49.333] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}" I[2021-03-01|12:41:49.333] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}" E[2021-03-01|12:41:49.414] dialing failed (attempts: 1): auth failure: secret conn failed: read tcp 142.93.45.202:51970->35.169.158.175:27146: read: connection reset by peer module=pex addr=fe5eb5a945598476abe4826a8d31b9f8da7b1a54@35.169.158.175:27146 I[2021-03-01|12:41:49.453] added a peer to state sync reactor module=statesync peer=b7affdf084115fd7e7cdbc04686ef8c218e1801c E[2021-03-01|12:41:49.491] dialing failed (attempts: 1): incompatible: Peer is on a different network. Got Binance-Chain-Ganges, expected Binance-Chain-Tigris module=pex addr=9612b570bffebecca4776cb4512d08e252119005@52.192.45.95:27146 E[2021-03-01|12:41:49.513] dialing failed (attempts: 1): incompatible: Peer is on a different network. Got Binance-Chain-Ganges, expected Binance-Chain-Tigris module=pex addr=8c379d4d3b9995c712665dc9a9414dbde5b30483@54.249.16.115:27146 I[2021-03-01|12:41:49.548] added a peer to state sync reactor module=statesync peer=5c96d4f0174dda7757bcda11a57ecc91851a90f0 E[2021-03-01|12:41:49.563] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex addr=1ebb7d9bc3ee3001f41200ed4f4be5432586c43e@34.92.4.164:27147 E[2021-03-01|12:41:49.563] dialing failed (attempts: 1): dial tcp 35.159.10.214:27146: i/o timeout module=pex addr=bfa682d7d5bb503b724f8dc34a0ef5f6bfa17d6f@35.159.10.214:27146 I[2021-03-01|12:41:49.817] added a peer to state sync reactor module=statesync peer=98579505572cc85488caa403bebb62e5b61d68e0 E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex addr=c52390d6073ea2b24de2cad787d012ce170bd5ac@34.92.4.164:27147 E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex addr=0f0804cba14f8ca80adb72e9a631658d344ad274@34.92.4.164:27147 E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 35.187.203.129:27147: i/o timeout module=pex addr=056d96d092fa3ee365e6bc8a17cf5115277c9b30@35.187.203.129:27147 E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 66.228.60.219:26656: i/o timeout module=pex addr=14e063f764661605de660fb3bca1d38f2ed70250@66.228.60.219:26656 E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 99.81.122.88:27146: i/o timeout module=pex addr=29792743565c4470480dbf7ef43b9256465dfa74@99.81.122.88:27146 E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex addr=b13390b1df3b8aff3e14e543f9bfea87e8db4745@34.92.4.164:27147 E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 18.139.29.55:27146: i/o timeout module=pex addr=6620efa13018c49857b626366ebd39c3f982c830@18.139.29.55:27146 E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 95.211.89.68:27146: i/o timeout module=pex addr=2abe967cad17cf179320ac31b7386ef5d73b6cef@95.211.89.68:27146 E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 82.65.68.233:27147: i/o timeout module=pex addr=6b8ac216bae99f92abd95787929911a651a1f028@82.65.68.233:27147 E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex addr=54609e59e4079e2cb33b96c821cb3df10485ac97@34.92.4.164:27147 I[2021-03-01|12:41:51.097] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 out}" I[2021-03-01|12:41:51.203] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}" E[2021-03-01|12:41:51.953] dialing failed (attempts: 1): auth failure: secret conn failed: read tcp 142.93.45.202:36030->18.208.56.183:27146: i/o timeout module=pex addr=c318936fc3005a07c332c282cd7c3f24fba0fb74@18.208.56.183:27146 I[2021-03-01|12:41:58.563] decided not init pool module=statesync manifests=0 tried=1 I[2021-03-01|12:41:58.563] broadcast state status request module=statesync I[2021-03-01|12:42:08.563] got enough peers to start state sync module=statesync manifestHash="[100 117 109 109 121 104 97 115 104 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]" numOfPeers=13 I[2021-03-01|12:42:08.563] all peers we connected have no manifest, start fast sync directly module=statesync panic: runtime error: invalid memory address or nil pointer dereference E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 82.65.68.233:27147: i/o timeout module=pex addr=6b8ac216bae99f92abd95787929911a651a1f028@82.65.68.233:27147 E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex addr=54609e59e4079e2cb33b96c821cb3df10485ac97@34.92.4.164:27147 I[2021-03-01|12:41:51.097] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 out}" I[2021-03-01|12:41:51.203] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}" E[2021-03-01|12:41:51.953] dialing failed (attempts: 1): auth failure: secret conn failed: read tcp 142.93.45.202:36030->18.208.56.183:27146: i/o timeout module=pex addr=c318936fc3005a07c332c282cd7c3f24fba0fb74@18.208.56.183:27146 I[2021-03-01|12:41:58.563] decided not init pool module=statesync manifests=0 tried=1 I[2021-03-01|12:41:58.563] broadcast state status request module=statesync I[2021-03-01|12:42:08.563] got enough peers to start state sync module=statesync manifestHash="[100 117 109 109 121 104 97 115 104 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]" numOfPeers=13 I[2021-03-01|12:42:08.563] all peers we connected have no manifest, start fast sync directly module=statesync panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xd42ef2] goroutine 155414 [running]: github.com/tendermint/tendermint/snapshot.(*StateReactor).startFastSync(0xc03743cd00) /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/snapshot/reactor.go:237 +0x222 github.com/tendermint/tendermint/snapshot.(*StateReactor).poolRoutine(0xc03743cd00) /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/snapshot/reactor.go:343 +0x26e created by github.com/tendermint/tendermint/snapshot.(*StateReactor).OnStart /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/snapshot/reactor.go:126 +0x6d ```

Can someone help me @huangsuyu ? I want to run an archive node but don't want to wait until the sync from scratch will finish, just need to sync from a snapshot.

alexqrid commented 3 years ago

Any updates on this? @huangsuyu Why does binance actually make the snapshots if they are useless as far as node can sync from them?

chainwhisper commented 3 years ago

@alexqrid Checking! i will come back to you asap

chainwhisper commented 3 years ago

Okay I just completed a sha256 hash of the compressed snapshot and it matches that checksum

can you check if you set open file limit to 65535?

alexqrid commented 3 years ago

can you check if you set open file limit to 65535?

Yes I did. Open file limit was set to 65535. The problem is with wrong block hash as you can see from the logs.

Logs ```sh I[2021-03-09|15:30:45.586] start to collect publish information module=pub height=142231940 I[2021-03-09|15:30:45.601] start to publish module=pub height=142231940 blockTime=1612418401567922666 numOfTrades=0 numOfOrders=7 numOfProposals=0 numOfSideProposals=0 numOfStakeUpdates=0 numOfAccounts=9 I[2021-03-09|15:30:45.601] Executed block module=state height=142231940 validTxs=8 invalidTxs=0 I[2021-03-09|15:30:45.613] Committed state module=state height=142231940 txs=8 appHash=8728829543CA547C713CEEEB1CAEA89447F7B11E1104460212099EA636B1C880 I[2021-03-09|15:30:45.620] failed to process NewOrderMsg module=main oid=DB5E68DFAD2F8CFBBFACC9C29266B95617A49FBC-375131 I[2021-03-09|15:30:45.620] symbols to match module=dexkeeper symbols=[RAVEN-F66_BNB] I[2021-03-09|15:30:45.621] Sync params proposals. module=paramHub I[2021-03-09|15:30:45.628] start to collect publish information module=pub height=142231941 E[2021-03-09|15:30:45.642] failed to locate order change in OrderChangesMap module=pub orderChange="id: DB5E68DFAD2F8CFBBFACC9C29266B95617A49FBC-375131, tpe: FailedBlocking" I[2021-03-09|15:30:45.646] start to publish module=pub height=142231941 blockTime=1612418401925314661 numOfTrades=0 numOfOrders=3 numOfProposals=0 numOfSideProposals=0 numOfStakeUpdates=0 numOfAccounts=3 I[2021-03-09|15:30:45.646] Executed block module=state height=142231941 validTxs=2 invalidTxs=1 I[2021-03-09|15:30:45.657] Committed state module=state height=142231941 txs=3 appHash=45E848914F26AE91F0F228C5225216B4879EE7FF976A26ECB49FF04B3DB52F02 panic: Failed to process committed block (142231942:9C1D7035617ACB4C3437C060F420419C283C2C5C49C5AA47083900FB52F851ED): Wrong Block.Header.AppHash. Expected 45E848914F26AE91F0F228C5225216B4879EE7FF976A26ECB49FF04B3DB52F02, got A3C8008FC6857F897CABA74860A7C8A1FFA7474739E8BA85E2721A3CC017E436 goroutine 155332 [running]: github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc000b10b60) /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/blockchain/v0/reactor.go:404 +0x194b created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/blockchain/v0/reactor.go:127 +0x84 ```
kenorb commented 3 years ago

See: https://github.com/cosmos/cosmos-sdk/issues/10281