search

Consensys / quorum

A permissioned implementation of Ethereum supporting data privacy
https://www.goquorum.com/
GNU Lesser General Public License v3.0
4.69k stars 1.3k forks source link

Block receipts missing, can't freeze (v21.4.0) #1382

Closed sokiaoba closed 2 years ago

sokiaoba commented 2 years ago

Currently I have a similar issue with https://github.com/ConsenSys/quorum/issues/1279.

Here is the log of the tx node.

/ # geth --datadir /data/dd
INFO [04-16|08:11:33.061] Starting Geth on Ethereum mainnet...
INFO [04-16|08:11:33.061] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [04-16|08:11:33.062] Maximum peer count                       ETH=50 LES=0 total=50
INFO [04-16|08:11:33.062] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[04-16|08:11:33.109] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:11:33.109] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:11:33.109] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:11:33.109] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:11:33.109] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:11:33.109] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
WARN [04-16|08:11:33.160] Found deprecated node list file /data/dd/geth/static-nodes.json, please use the TOML config file instead.
INFO [04-16|08:11:33.201] Set global gas cap                       cap=25000000
INFO [04-16|08:11:33.201] Allocated trie memory caches             clean=1023.00MiB dirty=1024.00MiB
INFO [04-16|08:11:33.207] Allocated cache and file handles         database=/data/dd/geth/chaindata cache=2.00GiB handles=524288
ERROR[04-16|08:11:34.110] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:11:34.110] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:11:34.110] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [04-16|08:11:35.074] Opened ancient database                  database=/data/dd/geth/chaindata/ancient
INFO [04-16|08:11:35.621] Initialised chain configuration          config="{ChainID: 10 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 IsQuorum: true Constantinople: 0 TransactionSizeLimit: 64 MaxCodeSize: 0 Petersburg: <nil> Istanbul: <nil>, Muir Glacier: <nil> YOLO v1: <nil> PrivacyEnhancements: <nil> Engine: istanbul}"
ERROR[04-16|08:11:35.630] Block receipts missing, can't freeze     number=201630 hash="7d656a…ba3c38"
INFO [04-16|08:11:35.742] Deep froze chain segment                 blocks=0 elapsed=183.079ms number=201629
INFO [04-16|08:11:35.764] Initialising Ethereum protocol           name=eth versions="[65 64 63]" network=1337 dbversion=8
INFO [04-16|08:11:35.764] Initialising Quorum consensus protocol   name=istanbul versions="[100 99 64]" network=1337 dbversion=8
INFO [04-16|08:11:35.810] Loaded most recent local header          number=3754207 hash="6a4523…588314" td=3754208 age=16h45m6s
INFO [04-16|08:11:35.810] Loaded most recent local full block      number=3754207 hash="6a4523…588314" td=3754208 age=16h45m6s
INFO [04-16|08:11:35.810] Loaded most recent local fast block      number=3754207 hash="6a4523…588314" td=3754208 age=16h45m6s
WARN [04-16|08:11:35.947] Head state missing, repairing            number=3754207 hash="6a4523…588314"

What I have done so far

  1. upgraded node version

Originally the error started to happen with quorum@21.1.0. I upgraded all nodes (1 tx node & 4 validator nodes) to quorum@21.4.0 but still have the same error.

  1. try to export & import blocks from other nodes

As mentioned in https://github.com/ConsenSys/quorum/issues/1279#issuecomment-976316645, I tried to export & import blocks from other validator nodes. But somehow geth export command seems to be hanging like below. 

/ # geth --datadir /data/dd export blocks 0 10
INFO [04-16|07:18:18.358] Maximum peer count                       ETH=50 LES=0 total=50
INFO [04-16|07:18:18.358] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[04-16|07:18:18.442] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|07:18:18.442] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|07:18:18.442] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|07:18:18.442] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|07:18:18.443] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|07:18:18.443] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
WARN [04-16|07:18:18.499] Found deprecated node list file /data/dd/geth/static-nodes.json, please use the TOML config file instead.
INFO [04-16|07:18:18.548] Set global gas cap                       cap=25000000
INFO [04-16|07:18:18.552] Allocated cache and file handles         database=/data/dd/geth/chaindata cache=512.00MiB handles=524288
ERROR[04-16|07:18:19.443] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|07:18:19.443] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|07:18:19.443] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [04-16|07:18:20.084] Opened ancient database                  database=/data/dd/geth/chaindata/ancient
INFO [04-16|07:18:20.335] Loaded most recent local header          number=3754207 hash="6a4523…588314" td=3754208 age=15h51m51s
INFO [04-16|07:18:20.335] Loaded most recent local full block      number=3754207 hash="6a4523…588314" td=3754208 age=15h51m51s
INFO [04-16|07:18:20.335] Loaded most recent local fast block      number=3754207 hash="6a4523…588314" td=3754208 age=15h51m51s
ERROR[04-16|07:18:20.370] Block receipts missing, can't freeze     number=201630  hash="7d656a…ba3c38"
WARN [04-16|07:18:20.418] Head state missing, repairing            number=3754207 hash="6a4523…588314"
INFO [04-16|07:18:20.480] Deep froze chain segment                 blocks=0 elapsed=184.483ms number=201629
  1. try to reset head block number

As mentioned in https://github.com/multi-geth/multi-geth/issues/134#issuecomment-741799884, I tried to reset head block number. But geth console command doesn't work in a kind of similar way with geth export.

/ # geth console --datadir /data/dd
INFO [04-16|08:07:06.865] Starting Geth on Ethereum mainnet...
INFO [04-16|08:07:06.865] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [04-16|08:07:06.866] Maximum peer count                       ETH=50 LES=0 total=50
INFO [04-16|08:07:06.866] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[04-16|08:07:06.896] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:07:06.896] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:07:06.896] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:07:06.896] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:07:06.896] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:07:06.896] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
WARN [04-16|08:07:06.921] Found deprecated node list file /data/dd/geth/static-nodes.json, please use the TOML config file instead.
INFO [04-16|08:07:06.956] Set global gas cap                       cap=25000000
INFO [04-16|08:07:06.956] Allocated trie memory caches             clean=1023.00MiB dirty=1024.00MiB
INFO [04-16|08:07:06.960] Allocated cache and file handles         database=/data/dd/geth/chaindata cache=2.00GiB handles=524288
ERROR[04-16|08:07:07.896] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:07:07.897] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[04-16|08:07:07.897] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [04-16|08:07:08.329] Opened ancient database                  database=/data/dd/geth/chaindata/ancient
INFO [04-16|08:07:08.552] Initialised chain configuration          config="{ChainID: 10 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 IsQuorum: true Constantinople: 0 TransactionSizeLimit: 64 MaxCodeSize: 0 Petersburg: <nil> Istanbul: <nil>, Muir Glacier: <nil> YOLO v1: <nil> PrivacyEnhancements: <nil> Engine: istanbul}"
ERROR[04-16|08:07:08.577] Block receipts missing, can't freeze     number=201630 hash="7d656a…ba3c38"
INFO [04-16|08:07:08.860] Initialising Ethereum protocol           name=eth versions="[65 64 63]" network=1337 dbversion=8
INFO [04-16|08:07:08.860] Initialising Quorum consensus protocol   name=istanbul versions="[100 99 64]" network=1337 dbversion=8
INFO [04-16|08:07:08.877] Deep froze chain segment                 blocks=0 elapsed=360.010ms number=201629
INFO [04-16|08:07:08.903] Loaded most recent local header          number=3754207 hash="6a4523…588314" td=3754208 age=16h40m39s
INFO [04-16|08:07:08.903] Loaded most recent local full block      number=3754207 hash="6a4523…588314" td=3754208 age=16h40m39s
INFO [04-16|08:07:08.903] Loaded most recent local fast block      number=3754207 hash="6a4523…588314" td=3754208 age=16h40m39s
WARN [04-16|08:07:08.956] Head state missing, repairing            number=3754207 hash="6a4523…588314"

System info

/ # geth version
Geth
Version: 1.9.20-stable
Git Commit: fb104840a007871394f645b9b0de8827833fb819
Quorum Version: 21.4.0
Architecture: amd64
Protocol Versions: [65 64 63]
Network Id: 1337
Go Version: go1.15.5
Operating System: linux
GOPATH=
GOROOT=go

Thank you.

baptiste-b-pegasys commented 2 years ago

Hello, thank you for raising this. I want to reproduce this, do you have any instructions or description to do so?

antonydenyer commented 2 years ago

I've tried a bunch of export and import combinations with various versions of GoQuorum and can not replicate this. I think the only possibility for us to debug would be to get a copy of the exported rlp.

antonydenyer commented 2 years ago

Unless we can reproduce this locally there's not much we can do, closing for now, feel free to re-open with steps to reproduce.