search

celo-org / celo-blockchain

Official repository for the golang Celo Blockchain
https://celo.org
GNU Lesser General Public License v3.0
560 stars 198 forks source link

Bad blocks syncing mainnet using master branch #2132

Closed jcortejoso closed 1 year ago

jcortejoso commented 1 year ago

Expected Behavior

master branch should be able to sync the hole chain for mainnet

Actual Behavior

Using master branch and golang 1.17.5, I get the next error:

ERROR[06-16|17:29:47.814] Error while calling engine.StopValidating err="stopped engine"
ERROR[06-16|17:29:52.550]
########## BAD BLOCK #########
Chain config: {ChainID: 42220 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Churrito: 6774000, Donut: 6774000, Espresso: 11838440, GFork: <nil>, Engine: istanbul}

Number: 815644
Hash: 0xa25dbcff718e240bf313c3f85a95648fc83ef8f3188831e3fa11cbd8a7660219
     0: cumulative: 125556 gas: 125556 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xf0cce1e8493c028816353e6a6dcb28d76adb027aef73e0333d50fd024b7d8ff2 logs: [0xc00036f760] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000002000004000000000000000000000000000001000000000000000008000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000100000000000000000000000000000000000000000002000000000000000000000000000000000000000220000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
     1: cumulative: 0 gas: 0 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0000000000000000000000000000000000000000000000000000000000000000 logs: [0xc00036f810] bloom: 00000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000200000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000 state:

Error: invalid merkle root (remote: 52e821675778c1e010211462fbe3b647596f4c35e474b37dccd963cd988f1616 local: e6ae89ff05dfc84b1c86478e49a4e346cae078ff25a9d28f71c5f094979f1510)
##############################

Steps to reproduce the behavior

  1. Start a full-node from scratch using master compilation
  2. Wait until it syncs 815643 blocks

System Information

Using docker image us.gcr.io/celo-testnet/geth:master

Chain/Network: Mainnet

Thanks to Don Patterson from Discord for reporting.

jcortejoso commented 1 year ago

More BAD BLOCKS:

Block 1209015:

ERROR[06-16|17:39:59.156] Error while calling engine.StopValidating err="stopped engine"
ERROR[06-16|17:39:59.638]
########## BAD BLOCK #########
Chain config: {ChainID: 42220 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Churrito: 6774000, Donut: 6774000, Espresso: 11838440, GFork: <nil>, Engine: istanbul}

Number: 1209015
Hash: 0x2a4798b09dd09eb16f1da1b63566609f665c096c45bdb5ec747dffb8ff491d51
     0: cumulative: 63831 gas: 63831 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x9eb553583dc26686e3f27c8472aca4214ed29d55ee368479ea138e0a71849b13 logs: [0xc001c99ad0] bloom: 00000000000000000000000000000000000000000000000000000000000000000000080000000000000000002000000000000000000000000000000000000000000000000000000000000008010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000001000000000000000000000000000000000000000000000000001000000000000000000000000000000800000000000000000000000000000000000000000000002000000000000000000000000000800040000000000000000000000000000010000000000000000000000000000000000000000000000000000000000 state:
     1: cumulative: 0 gas: 0 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0000000000000000000000000000000000000000000000000000000000000000 logs: [0xc001c99b80] bloom: 00000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000200000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000 state:

Error: invalid merkle root (remote: d71024feb689223ce969547f772e58e2a8496a5bab7f7b663bf220a2bdeef519 local: 7dbbb9325693c95e1a66d48e3b432fd58ca91753c6843ab567e3c1712af41f83)
##############################

WARN [06-16|17:39:59.639] Synchronisation failed, dropping peer    peer=fd94dd2596d9b518f40285cb53f9a7786bf394298092f3305754ff34dcd90663 err="retrieved hash chain is invalid: invalid merkle root (remote: d71024feb689223ce969547f772e58e2a8496a5bab7f7b663bf220a2bdeef519 local: 7dbbb9325693c95e1a66d48e3b432fd58ca91753c6843ab567e3c1712af41f83)"
ERROR[06-16|17:39:59.639] Error while calling engine.StopValidating err="stopped engine"

Block 2159017:

ERROR[06-16|17:40:46.159] Ethereum peer removal failed             peer=16e3f4dd err="peer not registered"
ERROR[06-16|17:40:47.972]
########## BAD BLOCK #########
Chain config: {ChainID: 42220 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Churrito: 6774000, Donut: 6774000, Espresso: 11838440, GFork: <nil>, Engine: istanbul}

Number: 2159017
Hash: 0x18009b1a0945725a422f0df0cc7ad062cee8f0f178356f1c01c0c8f4bdc7f37c
     0: cumulative: 48831 gas: 48831 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xeb0c158b03b660aefa166e19fdb1673e37f6be65381d6c54808f3f94d45e0245 logs: [0xc00f304840] bloom: 00000000000000000000000000000000000000000000000000000000000000000000080000000000000000002000000000000000000000000000000000000000000000000000000000000008010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000001000000000000000000000000000000000000000000000000001000000000000000000000000000000800000000000000000000000000000000000000000000002000000000000000000000000000800040000000000000000000000000000010000000000000000000000000000000000000000000000000000000000 state:
     1: cumulative: 0 gas: 0 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0000000000000000000000000000000000000000000000000000000000000000 logs: [0xc00f3048f0] bloom: 00000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000200000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000 state:

Error: invalid merkle root (remote: 675e7b2aab902bfc19d4f7221dfff3fd69f75a4920ccdb6f18d83820f2b511dc local: 81a305b89b3577bb9e42e7bed84f3453ae6531104a439838a44fcc30dceadf74)
##############################

WARN [06-16|17:40:47.972] Synchronisation failed, dropping peer    peer=37d563a3eb4d34569c5ba380e4fd8681683b0fc1d5a0a13ad3026a9ef79d9e47 err="retrieved hash chain is invalid: invalid merkle root (remote: 675e7b2aab902bfc19d4f7221dfff3fd69f75a4920ccdb6f18d83820f2b511dc local: 81a305b89b3577bb9e42e7bed84f3453ae6531104a439838a44fcc30dceadf74)"
ERROR[06-16|17:40:47.972] Error while calling engine.StopValidating err="stopped engine"
ERROR[06-16|17:40:47.972] Ethereum peer removal failed             peer=37d563a3 err="peer not registered"

Block 3255621:

ERROR[06-16|17:41:01.375] Error while calling engine.StopValidating err="stopped engine"
ERROR[06-16|17:41:01.545]
########## BAD BLOCK #########
Chain config: {ChainID: 42220 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Churrito: 6774000, Donut: 6774000, Espresso: 11838440, GFork: <nil>, Engine: istanbul}

Number: 3255621
Hash: 0xe015922be8f7507a9c5abd2587ed10e165eaec34c158e929e4ef4edae30df603
     0: cumulative: 63831 gas: 63831 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xf41d80ab599c80e983d0bafcfdb5287a3e24a16fff1e70fa4eedd4e7bcb6c6d1 logs: [0xc0102238c0] bloom: 00000000000000000000000000000008000000000000000000000000000000000000080000000000000000002000000000000000000000000000000000000000000000000000000000000008010000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000002000000000100000000000000000800040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
     1: cumulative: 306583 gas: 242752 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x9d16831cc8ba2a3fbde00e0a7302f57bdcb3195ff17b6ce87cb856c40fc4f29c logs: [0xc010223970 0xc010223a20] bloom: 00800000000000000000200000000000000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000200000000000000000000000000000000004000040000000000000000000000000000000000000400000008400040000000000000000000000000000000000000000000000401000000000000800000000000000000000000000000000000000800000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
     2: cumulative: 0 gas: 0 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0000000000000000000000000000000000000000000000000000000000000000 logs: [0xc010223b80] bloom: 00000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000200000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000 state:

Error: invalid merkle root (remote: bb6661e792f9534a37087a0e1bf5291de9af846021e32ccd8611fa391d784f97 local: 2601087b1bf0b6ad57d2e2bb79e0ee53371fba9eed44e8fabaa92a5cbc71e461)
##############################

WARN [06-16|17:41:01.545] Synchronisation failed, dropping peer    peer=6c5d213891e938ee8892b09b896537b70f2d42104843e87d52e0db3bfbab7a7f err="retrieved hash chain is invalid: invalid merkle root (remote: bb6661e792f9534a37087a0e1bf5291de9af846021e32ccd8611fa391d784f97 local: 2601087b1bf0b6ad57d2e2bb79e0ee53371fba9eed44e8fabaa92a5cbc71e461)"
ERROR[06-16|17:41:01.545] Error while calling engine.StopValidating err="stopped engine"
jcortejoso commented 1 year ago

Block 4126439

ERROR[06-16|17:42:37.131] Error while calling engine.StopValidating err="stopped engine"
ERROR[06-16|17:42:38.301]
########## BAD BLOCK #########
Chain config: {ChainID: 42220 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Churrito: 6774000, Donut: 6774000, Espresso: 11838440, GFork: <nil>, Engine: istanbul}

Number: 4126439
Hash: 0x90f15c128e529d78f7175da8676d8c86ac162b7b27cd7e7fe1c3ab23f0a19cc2
     0: cumulative: 94631 gas: 94631 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7da62cacae9ea09d3a8e343179dbc0dbdbd035bf13489aaf1a472d45d31c8cf4 logs: [0xc013004370 0xc013004420 0xc0130044d0 0xc013004580] bloom: 0000000000000000000000000000000000000000000000000000000000000008000008000000000000000000a00000000000000200000000000000000000000000000010000000000000000800000000000000000000000000000000000000004000000000000600000000200000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000800000000000000000000000000000000040000000000000200000000000000000000000000080000000000000000000000000000000000000000000000000c000200000000000000000000000000000200000000 state:
     1: cumulative: 0 gas: 0 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0000000000000000000000000000000000000000000000000000000000000000 logs: [0xc013004630] bloom: 00000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000200000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000 state:

Error: invalid merkle root (remote: 8171d4cd604eda641825d6485f2751485d6c12d737b5f471aa58630371f0ded8 local: 22ea019341eb847c7899f07ee0165f29287b70ba55899d2c94744393a24361d6)
##############################

WARN [06-16|17:42:38.301] Synchronisation failed, dropping peer    peer=7062d4dff3f7ef987f5fbab4afaf3aee792802ba36952d77a7c1f6bfee2e0e77 err="retrieved hash chain is invalid: invalid merkle root (remote: 8171d4cd604eda641825d6485f2751485d6c12d737b5f471aa58630371f0ded8 local: 22ea019341eb847c7899f07ee0165f29287b70ba55899d2c94744393a24361d6)"
ERROR[06-16|17:42:38.301] Error while calling engine.StopValidating err="stopped engine"

Block 5235521:

ERROR[06-16|17:43:17.351] Error while calling engine.StopValidating err="stopped engine"
ERROR[06-16|17:43:25.699]
########## BAD BLOCK #########
Chain config: {ChainID: 42220 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Churrito: 6774000, Donut: 6774000, Espresso: 11838440, GFork: <nil>, Engine: istanbul}

Number: 5235521
Hash: 0xedfc4ce14a7c6c404becb2e391aa3aafd5ef4bcf5db6e4204010349d85b96ce0
     0: cumulative: 46144 gas: 46144 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xed34034fa7d9ac4748c9153cc65d7cc9e278038406667155e0c72d3f99dc5273 logs: [0xc0237c1c30] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000800000000000000000008000000000000000000000000000000000000000000000400000000000000000040000000000000000000000010000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000000000000000000002000000000000000000000000000000000000000000000000000000000008000008000000000000000000000000000000000000000000000004000000 state:
     1: cumulative: 118860 gas: 72716 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xba3fe0938ee723d214eb4212723bb2e37cefb55d8b77678ee0e80883676cc2fc logs: [0xc0237c1d90 0xc0237c1e40] bloom: 00000000000000000000000000000000002000000000000000000000000000000000080000000000000000002000000000000000000000000002000000240000000000000100000000000000000000000800000000060000000000800000000000000000000000000000000000000400000000000000000000000000000000000100000000000000001000000000000000000000000000000000000000000000020000000000000100008000000000000000000000000000000000000000000000000000000200000000000000000000000800000040000000000000000040000010000080000000000000000000000000000000000000000000000000000000 state:
     2: cumulative: 235125 gas: 116265 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xcf7d759aa3a78705f50be51d560eba1743e4e71701cadc35414da3cd87b18753 logs: [0xc0237c1ef0 0xc01acd0000 0xc01acd00b0 0xc01acd0160] bloom: 00000000000000000000000000000000000000000000000000000000000000000000080000000041000000002000000000000000000000000000000000000000000000100000000000000008000040000000200000000000000000000000000000000000000000000000000000000000000000000000100000100010000000000000000000000000000000000000000000000000000004000100000004008000000000000000000000000000000000000000000000000000000000000400000000000002000800020000000000000000000800000000000000000004000000000800000000000000000004000000000000000000000000000000200000000000 state:
     3: cumulative: 342387 gas: 107262 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x9eed6612d77bb07cfdd5a775d2167a6b1ae569c8552667c7540c001628616a11 logs: [0xc01acd0210 0xc01acd02c0 0xc01acd0370 0xc01acd0420] bloom: 00000000000000000000000000000020000000000000000000000000100000000000080000000040000000002000000000000000000000000000000000000000000000100000000000000008000000000000000000000000000000000000000080000000100000080000000000000000000000000000100000000010000000000000000000000000000000000000000000000000000004040000000000000000000000000000000000000000000200000000000000000000000000000400000000000002000000000000000000000000000800000000000000000000000002000000000000000000000004000800000000000000000000000000000000000000 state:
     4: cumulative: 0 gas: 0 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0000000000000000000000000000000000000000000000000000000000000000 logs: [0xc01543c4d0] bloom: 00000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000200000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000 state:

Error: invalid merkle root (remote: e8c588ba9fcbce28b7afc6fd4fda0b04b9e71e51df1b74cab587d0f4b91581f4 local: a365aa793c8100b073baa762881ff475ebe70a8b6def7acf8b8145e7b57c5220)
##############################

WARN [06-16|17:43:25.716] Synchronisation failed, dropping peer    peer=6853a6f948ba30c7f30b27d771d22e1b1a27b81429304feab33a23db545cc502 err="retrieved hash chain is invalid: invalid merkle root (remote: e8c588ba9fcbce28b7afc6fd4fda0b04b9e71e51df1b74cab587d0f4b91581f4 local: a365aa793c8100b073baa762881ff475ebe70a8b6def7acf8b8145e7b57c5220)"
ERROR[06-16|17:43:25.717] Error while calling engine.StopValidating err="stopped engine"

Block 6023364:

ERROR[06-16|17:44:11.976] Error while calling engine.StopValidating err="stopped engine"
ERROR[06-16|17:44:17.428]
########## BAD BLOCK #########
Chain config: {ChainID: 42220 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Churrito: 6774000, Donut: 6774000, Espresso: 11838440, GFork: <nil>, Engine: istanbul}

Number: 6023364
Hash: 0x396d0d3fb76c9fce3b45407f3656f35f7223d52ed196b011bc42730cde8b70fa
     0: cumulative: 46144 gas: 46144 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x09783d61e7902b5d49f700895980babea273d51726414cdb0fc75b72d7c051ea logs: [0xc003bf93f0] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000400000000000000000040000000004000000008000010000010000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000 state:
     1: cumulative: 162431 gas: 116287 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x215fb0c7ae6c1f6604d91b2f49f6547d1dea1d7dbdd45b51e974f4de4815cbae logs: [0xc003bf9550 0xc003bf9600 0xc003bf96b0 0xc003bf9760] bloom: 00000000000004000000800000000000000000000000000000000000000000000000080000000000000000002000000000000000000000000000001000000000000000100000000000000008000000000000000000000000000000000000000000000000000004000000100000000000000000000000000000100010000000000000000008000000000000000800000000000000000000000000000000008000000000020000000020000000000000000000000000000000000000000400000000004002000000020000000000000000000800010000000000000000000000000000000000000000000004000000000000000000000000000000000000000000 state:
     2: cumulative: 0 gas: 0 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0000000000000000000000000000000000000000000000000000000000000000 logs: [0xc003bf9810] bloom: 00000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000200000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000 state:

Error: invalid merkle root (remote: f9824822f2ab2f019e7c1a76d09845de3a0d2af1798c6b113945fda2de7568f3 local: 51f5d8f8d919cfcc950735a28e6389465b7a5bb782bcbb68e5bf7207f53e6c0e)
##############################

WARN [06-16|17:44:17.428] Synchronisation failed, dropping peer    peer=f7319107b100c88d189f1db67c7e8baa69691f1022aca6b55cd1ac2cf152c690 err="retrieved hash chain is invalid: invalid merkle root (remote: f9824822f2ab2f019e7c1a76d09845de3a0d2af1798c6b113945fda2de7568f3 local: 51f5d8f8d919cfcc950735a28e6389465b7a5bb782bcbb68e5bf7207f53e6c0e)"

Block 7957292:

ERROR[06-16|17:44:31.904] Error while calling engine.StopValidating err="stopped engine"
ERROR[06-16|17:44:31.904] Ethereum peer removal failed             peer=f4fcc73a err="peer not registered"
ERROR[06-16|17:44:33.816]
########## BAD BLOCK #########
Chain config: {ChainID: 42220 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Churrito: 6774000, Donut: 6774000, Espresso: 11838440, GFork: <nil>, Engine: istanbul}

Number: 7957292
Hash: 0x3905c512d76df599e85b45a016db6ea46c851e29ac2798bf123e2f2449f93d21
     0: cumulative: 182847 gas: 182847 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x48445b10dd1b9a5d6ca866a5b981a85d35a8f60f8af097a3ae2fb24a52750359 logs: [0xc00fd82e70 0xc00fd82f20 0xc00fd82fd0 0xc00fd83080] bloom: 00000400000000000000000000000000000000000000000000000000000000000000080000000000000000402000000000000000000000000000000000000000000000100400000040800008000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000080000000008000000000400000000000002000000000000000000000000000800000040000000000000000000000000000000000000000004001000000000000000000000000000004000000000 state:
     1: cumulative: 238290 gas: 55443 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x58cea269288dceb3f02d4c7176ca9b906b01f43a7b42368558d165c0b04ee332 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
     2: cumulative: 293703 gas: 55413 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x2048f05b5433a2c3dd0ec8c456cbf48058da7c78f559e168c0804d83746efaba logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
     3: cumulative: 0 gas: 0 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0000000000000000000000000000000000000000000000000000000000000000 logs: [0xc00fd83130] bloom: 00000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000200000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000 state:

Error: invalid merkle root (remote: a156816150ee81568a590266e6dc7eaa5260eb658b7b1af32d0ce504bb91d21c local: be399fef9040d8cf9ada54c0d0fcf31e985919d85cd83b5cb4d0cdf6fcc977a7)
##############################

WARN [06-16|17:44:33.816] Synchronisation failed, dropping peer    peer=fd091df7d444a0577d3676851e2d49d94f25196c292bd5fab05b42f926aa3632 err="retrieved hash chain is invalid: invalid merkle root (remote: a156816150ee81568a590266e6dc7eaa5260eb658b7b1af32d0ce504bb91d21c local: be399fef9040d8cf9ada54c0d0fcf31e985919d85cd83b5cb4d0cdf6fcc977a7)"
djp3 commented 1 year ago

Thanks @jcortejoso . Monitoring for the fix.

aaronmboyd commented 1 year ago

Blocks 18607340, 18621934 also.

palango commented 1 year ago

Bisecting the problem on block 815644 gave me the following commit as the culprit:

c8843c1c80910494dba8d7075213fa2933ccd91e is the first bad commit

But I haven't yet looked what exactly goes wrong there.

git bisect log:

% git bisect log
# bad: [8760eea6d0605f148002b6bce1aedc2fe30e9f50] Cleanup Precompiles (#2092) (#2103)
# good: [ebe6ae76fddae9191391636d190b36d400c04583] Stricter input legth checking in precompiles getValidator/fractionMulExps/transfer (#2038)
git bisect start '8760eea6d' 'ebe6ae76f'
# good: [8652094cf95f18a8142f57d85d4a8eaa3d19fb78] Fix build after removing `clef` (#2091)
git bisect good 8652094cf95f18a8142f57d85d4a8eaa3d19fb78
# bad: [ec09ad80d9675daaacd6faa2277f46d62f949f7b] Apply `go1.18.10 fmt ./...` (#2107)
git bisect bad ec09ad80d9675daaacd6faa2277f46d62f949f7b
# good: [8209760ce346e8373d559399baa2db431dda5b93] Sign and publish to celo-org in GH Actions (#2101)
git bisect good 8209760ce346e8373d559399baa2db431dda5b93
# bad: [33359a9f8ad4ebb028ed85d1611812d067193c9d] Update ethers to to ethers 6 (#2081)
git bisect bad 33359a9f8ad4ebb028ed85d1611812d067193c9d
# bad: [c8843c1c80910494dba8d7075213fa2933ccd91e] Cleanup/tx processing (#2102)
git bisect bad c8843c1c80910494dba8d7075213fa2933ccd91e
# first bad commit: [c8843c1c80910494dba8d7075213fa2933ccd91e] Cleanup/tx processing (#2102)
aaronmboyd commented 1 year ago

Are we to assume 1.7.4 is unstable at this point or is it OK to run?

palango commented 1 year ago

Are we to assume 1.7.4 is unstable at this point or is it OK to run?

All releases are stable and (should) sync without problems. This bug only refers to the unstable master branch, which you shouldn't never use in production.

@aaronmboyd If you encounter problems with any release (as per the discord message), please open a new issue for that with all possible information. Thank you.

aaronmboyd commented 1 year ago

@palango Thanks, I have already #2134

hbandura commented 1 year ago

confirmed the commit that @palango mentioned. I did a manual binary search testing syncs and got to the same result. I was a bit slowed down by another issue we have in master that was fixed before this.

I haven't been able to pinpoint the exact issue in the commit that's the culprit yet, but I created a new branch from current master, reverting that commit, and the sync doesn't face that issue.

hbandura commented 1 year ago

I've just synced from scratch with the latest master ( e10394dfcaef2825ca799c97ddea3d3e78593724 )to 900k blocks without issues. I'll leave it longer but it seems that the previous problem was solved.

palango commented 1 year ago

The cause for this has been reverted in https://github.com/celo-org/celo-blockchain/pull/2145.