The Round Change continues indefinitely, BUT no new block is being generated.

Mameta29 commented 11 months ago

I built a GoQuorum chain on AWS's EKS service following this tutorial: https://docs.goquorum.consensys.net/tutorials/kubernetes/deploy-charts

Overview:

I was able to operate it for about a month, but suddenly blocks could no longer be produced. I was using EC2 instances for the EKS nodes, and their memory and CPU limits had been exceeded. I adjusted these and prepared sufficient EC2 instances. However, the same issue persists.

Regarding the event, a Round Change is taking place and an attempt is being made to produce blocks. There are no errors in each of the steps, but block production is not being reached. (I've posted the logs from the validators below.)

Validators: 4 RPC: 1

I am currently stuck as I cannot determine the cause and solution. Any assistance would be greatly appreciated.

goquorum-node-validator-4-quorum WARN [09-29|07:08:39.791] QBFT: TIMER CHANGING ROUND               address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=6 current.sequence=51045 state="Accept request" pr=<nil>
goquorum-node-validator-4-quorum INFO [09-29|07:08:39.791] QBFT: initialize new round               address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=6 current.sequence=51045 target.round=7 lastProposal.number=51044 lastProposal.hash=474faa..cfa230
goquorum-node-validator-4-quorum DEBUG[09-29|07:08:39.791] QBFT: lookup for pending block proposal requests address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-quorum TRACE[09-29|07:08:39.791] QBFT: start new ROUND-CHANGE timer       address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state="Accept request" timeout=1280.000
goquorum-node-validator-4-quorum INFO [09-29|07:08:39.791] QBFT: start new round                    address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 old.round=6  old.sequence=51045 old.state="Accept request" old.proposer=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 next.round=7 next.seq=51045 next.proposer=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 next.valSet="[0x2839A248bAA592B8fA4E566e1585493925A2F8A0 0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A 0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7]" next.size=4 next.IsProposer=false
goquorum-node-validator-4-quorum WARN [09-29|07:08:39.791] QBFT: TIMER CHANGED ROUND                address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=6 current.sequence=51045 state="Accept request" pr=<nil>
goquorum-node-validator-4-quorum INFO [09-29|07:08:39.791] QBFT: broadcast ROUND-CHANGE message     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x0000000000000000000000000000000000000000 msg.round=7 msg.sequence=51045 payload=0xf84df849c582c76507c0b84133be26a85f57125f985b9fc4670356e18a7b6205fe35422ff2d53e86c60a81e131f2d54dcbea2fed475bb4859bf0d3f68428072408ee7306bc6db6ccd237d07a00c0c0
goquorum-node-validator-4-quorum INFO [09-29|07:08:39.792] QBFT: handle ROUND-CHANGE message        address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=7 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=0
goquorum-node-validator-4-quorum TRACE[09-29|07:08:39.792] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-quorum TRACE[09-29|07:08:39.792] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-quorum DEBUG[09-29|07:08:39.792] QBFT: accepted ROUND-CHANGE messages     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=7 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=1
goquorum-node-validator-1-quorum DEBUG[09-29|07:08:39.792] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-3-quorum DEBUG[09-29|07:08:39.792] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-quorum DEBUG[09-29|07:08:39.793] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-3-quorum DEBUG[09-29|07:08:39.794] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-quorum DEBUG[09-29|07:08:39.794] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-1-quorum INFO [09-29|07:08:39.793] QBFT: handle ROUND-CHANGE message        address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=5 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=7 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=1
goquorum-node-validator-1-quorum TRACE[09-29|07:08:39.793] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=5 current.sequence=51045 state="Accept request"
goquorum-node-validator-1-quorum TRACE[09-29|07:08:39.793] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=5 current.sequence=51045 state="Accept request"
goquorum-node-validator-1-quorum DEBUG[09-29|07:08:39.793] QBFT: accepted ROUND-CHANGE messages     address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=5 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=7 msg.sequence=51045 higherRoundChanges.count=1 currentRoundChanges.count=1
goquorum-node-validator-2-quorum DEBUG[09-29|07:08:51.738] Current full block not old enough        number=51044 hash=5db7a5..307312 delay=3,162,240
goquorum-node-validator-2-quorum DEBUG[09-29|07:08:53.361] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-quorum DEBUG[09-29|07:08:55.393] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-1-quorum DEBUG[09-29|07:08:55.763] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-3-quorum DEBUG[09-29|07:08:56.235] Current full block not old enough        number=51044 hash=5db7a5..307312 delay=3,162,240

joshuafernandes commented 11 months ago

Hello @Mameta29

To confirm the above:

Your setup is on EKS and you have GoQ
You have had blocks being produced and it has suddenly stopped producing blocks?
All you see now is round changes?

Could you put some logs of the time this happened? Also what changed in your infrastructure/versions etc around the time that the blocks stopped?

Typically that would indicate one of the following:

that the keys (consequently the addresses) have changed and the expected validators are no longer operating
peering is somehow lost so the required number of validators cannot agree on a block
nodes were shutdown for a long period of time

Steps to resolve the above are:

verify each of the validators have the correct nodekeys
restart all of the validators in the pool sequentially (this is done to negate the block time intervals ie If more than 1/3 of validators stop participating, new blocks can no longer be created and requesttimeoutseconds doubles with each round change. The quickest method to resume block production is to restart all validators, which resets requesttimeoutseconds to its genesis value. https://docs.goquorum.consensys.net/configure-and-manage/configure/consensus-protocols/qbft#block-time

Mameta29 commented 11 months ago

@joshuafernandes Thank you for your response.

You have had blocks being produced and it has suddenly stopped producing blocks?

-> YES

All you see now is round changes?

-> I see round changes, when I send a tx. Nomally there is nothing change like below.

goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[10-03|04:40:24.634] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:40:25.761] Transaction pool status report           executable=2 queued=0 stales=0
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:40:26.073] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:40:26.421] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:40:27.749] Transaction pool status report           executable=2 queued=0 stales=0
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:40:28.302] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:40:44.103] Current full block not old enough        number=51044 hash=5db7a5..307312 delay=3,162,240
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[10-03|04:40:44.634] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:40:46.074] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:40:46.422] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:40:46.560] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:40:48.302] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:40:59.452] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[10-03|04:41:04.635] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:41:06.074] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:41:06.423] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:41:08.303] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[10-03|04:41:18.006] Current full block not old enough        number=51044 hash=5db7a5..307312 delay=3,162,240
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[10-03|04:41:24.636] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:41:26.075] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:41:26.424] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:41:28.303] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:41:44.103] Current full block not old enough        number=51044 hash=5db7a5..307312 delay=3,162,240
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[10-03|04:41:44.636] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:41:46.075] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:41:46.425] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:41:46.561] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:41:48.304] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:41:59.453] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[10-03|04:42:04.637] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:42:06.077] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:42:06.425] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:42:08.304] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[10-03|04:42:18.007] Current full block not old enough        number=51044 hash=5db7a5..307312 delay=3,162,240
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[10-03|04:42:24.638] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:42:26.077] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[10-03|04:42:26.426] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:42:28.305] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s

About steps:

I'm sorry to ask such a question, but how can I verify if each node key is correct?
I tried restarting all validators one by one. In this environment (as developed by that tutorial), I used 'helm uninstall' for it. Should I restart them all 'at the same time' instead of 'one by one'?

Mameta29 commented 11 months ago

Information added;

kubernetest version 1.26
github Tags:0.1.1 https://github.com/Consensys/quorum-kubernetes

joshuafernandes commented 11 months ago

Hello @Mameta29

Yep that makes sense if the validators were uninstalled.

To check on the validator keys you can check the secrets - each secret should have nodekey(or privatekey), public key and address and they get mounted in each container as files. The keys and such for the validators get created as part of the genesis chart. You then need to check the genesis file itself and see what addresses the genesis is expecting in the extraData ie each validators address will be part of the extra data field (https://besu.hyperledger.org/private-networks/how-to/configure/consensus/clique#extra-data)

It is possible depending on the values.yml overrides the keys were deleted and new keys would have been created when your validators were re installed - those keys will not match whats in the genesis extra data and hence you get no blocks created. This part we are careful off and from memory we don't delete the keys because once the nodeprivate key is delete you can't create it again. To override this behavior the values.yml change this flag https://github.com/Consensys/quorum-kubernetes/blob/master/helm/values/validator.yml#L5

Mameta29 commented 11 months ago

@joshuafernandes

Thank you for your response. The concern is that when launching a validator, there is a possibility that the keys do not match.

However, when the block generation stopped, there were no specific changes made to the validator, and it suddenly stopped. The attempt to restart the validator with 'helm uninstall' was done after the block generation had already stopped.

Also, when reviewing the 'values.yml' file, it was configured as follows:

For the four validators: removeKeysOnDelete: false
For the RPC node: removeKeysOnDelete: true

To confirm whether the keys necessary for block generation are mounted correctly within the Kubernetes pods for both validators and the RPC node, you can check the 'keys' directory within each pod. If the keys are correctly mounted, you should be able to see them in the respective 'keys' directories.

Regarding the absence of the genesis file as a pod, it's not typically a pod itself. The genesis file is usually used during the initialization of the network and should be accessible on the file system of the nodes, often outside of the pods.

mameta ~ %kubectl get pods --all-namespaces           
NAMESPACE     NAME                                                              READY   STATUS    RESTARTS   AGE
kube-system   aws-node-2fchs                                                    1/1     Running   0          4d1h
kube-system   aws-node-7l7c2                                                    1/1     Running   0          27h
kube-system   aws-node-7qj4b                                                    1/1     Running   0          27h
kube-system   aws-node-j2zp9                                                    1/1     Running   0          4d21h
kube-system   aws-node-j5ldk                                                    1/1     Running   0          27h
kube-system   aws-node-n669k                                                    1/1     Running   0          27h
kube-system   aws-node-w89tm                                                    1/1     Running   0          4d20h
kube-system   coredns-6f95bf5c49-76dp6                                          1/1     Running   0          3d23h
kube-system   coredns-6f95bf5c49-cl4rq                                          1/1     Running   0          3d23h
kube-system   csi-secrets-store-provider-aws-2bzj2                              1/1     Running   0          27h
kube-system   csi-secrets-store-provider-aws-5lz2r                              1/1     Running   0          27h
kube-system   csi-secrets-store-provider-aws-b2knp                              1/1     Running   0          4d20h
kube-system   csi-secrets-store-provider-aws-ljffx                              1/1     Running   0          4d20h
kube-system   csi-secrets-store-provider-aws-nnf8t                              1/1     Running   0          27h
kube-system   csi-secrets-store-provider-aws-pg4lp                              1/1     Running   0          4d1h
kube-system   csi-secrets-store-provider-aws-slltf                              1/1     Running   0          27h
kube-system   csi-secrets-store-secrets-store-csi-driver-7h7cp                  3/3     Running   0          27h
kube-system   csi-secrets-store-secrets-store-csi-driver-hm6vq                  3/3     Running   0          27h
kube-system   csi-secrets-store-secrets-store-csi-driver-j7qnd                  3/3     Running   0          4d20h
kube-system   csi-secrets-store-secrets-store-csi-driver-sjpmj                  3/3     Running   0          4d1h
kube-system   csi-secrets-store-secrets-store-csi-driver-tr6fw                  3/3     Running   0          27h
kube-system   csi-secrets-store-secrets-store-csi-driver-xxb6l                  3/3     Running   0          4d21h
kube-system   csi-secrets-store-secrets-store-csi-driver-zbk5h                  3/3     Running   0          27h
kube-system   ebs-csi-controller-669d6cb99-cc7qz                                6/6     Running   0          3d23h
kube-system   ebs-csi-controller-669d6cb99-ls7t2                                6/6     Running   0          3d23h
kube-system   ebs-csi-node-4jznc                                                3/3     Running   0          27h
kube-system   ebs-csi-node-6zn9z                                                3/3     Running   0          27h
kube-system   ebs-csi-node-bvkzn                                                3/3     Running   0          4d20h
kube-system   ebs-csi-node-hh96j                                                3/3     Running   0          4d21h
kube-system   ebs-csi-node-sr2q5                                                3/3     Running   0          27h
kube-system   ebs-csi-node-wptjf                                                3/3     Running   0          27h
kube-system   ebs-csi-node-wrnkk                                                3/3     Running   0          4d1h
kube-system   kube-proxy-99cdf                                                  1/1     Running   0          27h
kube-system   kube-proxy-9cz5q                                                  1/1     Running   0          27h
kube-system   kube-proxy-cv2s8                                                  1/1     Running   0          4d1h
kube-system   kube-proxy-jqffj                                                  1/1     Running   0          27h
kube-system   kube-proxy-ndrbs                                                  1/1     Running   0          27h
kube-system   kube-proxy-sr54z                                                  1/1     Running   0          4d20h
kube-system   kube-proxy-xd78t                                                  1/1     Running   0          4d21h
ryukyu        alertmanager-monitoring-kube-prometheus-alertmanager-0            2/2     Running   0          3d23h
ryukyu        goquorum-node-rpc-1-0                                             1/1     Running   0          3d23h
ryukyu        goquorum-node-validator-1-0                                       1/1     Running   0          4d1h
ryukyu        goquorum-node-validator-2-0                                       1/1     Running   0          3d23h
ryukyu        goquorum-node-validator-3-0                                       1/1     Running   0          3d23h
ryukyu        goquorum-node-validator-4-0                                       1/1     Running   0          3d23h
ryukyu        monitoring-grafana-57b7ffccb-qzgl6                                3/3     Running   0          3d23h
ryukyu        monitoring-kube-prometheus-operator-668f7945cf-mml4h              1/1     Running   0          3d23h
ryukyu        monitoring-kube-state-metrics-5979b7b8f6-gz8jf                    1/1     Running   0          4d1h
ryukyu        monitoring-prometheus-node-exporter-45z8n                         1/1     Running   0          27h
ryukyu        monitoring-prometheus-node-exporter-522g9                         1/1     Running   0          4d1h
ryukyu        monitoring-prometheus-node-exporter-7rh6z                         1/1     Running   0          27h
ryukyu        monitoring-prometheus-node-exporter-8twjl                         1/1     Running   0          27h
ryukyu        monitoring-prometheus-node-exporter-jb8ng                         1/1     Running   0          27h
ryukyu        monitoring-prometheus-node-exporter-kf5kl                         1/1     Running   0          4d21h
ryukyu        monitoring-prometheus-node-exporter-lcfnm                         1/1     Running   0          4d20h
ryukyu        prometheus-monitoring-kube-prometheus-prometheus-0                2/2     Running   0          3d23h
ryukyu        quorum-monitoring-ingress-ingress-nginx-controller-579d499fpkkt   1/1     Running   0          4d1h
ryukyu        quorum-network-ingress-ingress-nginx-controller-64d7fc894d7wsnb   1/1     Running   0          3d23h

I appreciate your patience, and please feel free to ask any further questions.

Mameta29 commented 11 months ago

I add some information. I found all validators and a rpc in the /config/static/static-nodes.json

/config/static # cat static-nodes.json
[
  "enode://157c<...some letters...>@goquorum-node-validator-4-0.goquorum-node-validator-4.ryukyu.svc.cluster.local:30303?discport=0",
  "enode://6578<...some letters...>@goquorum-node-validator-1-0.goquorum-node-validator-1.ryukyu.svc.cluster.local:30303?discport=0",
  "enode://c89b<...some letters...>@goquorum-node-rpc-1-0.goquorum-node-rpc-1.ryukyu.svc.cluster.local:30303?discport=0",
  "enode://effe<...some letters...>@goquorum-node-validator-3-0.goquorum-node-validator-3.ryukyu.svc.cluster.local:30303?discport=0",
  "enode://ffb7<...some letters...>@goquorum-node-validator-2-0.goquorum-node-validator-2.ryukyu.svc.cluster.local:30303?discport=0"
]

joshuafernandes commented 11 months ago

Hi @Mameta29

However, when the block generation stopped, there were no specific changes made to the validator, and it suddenly stopped. The attempt to restart the validator with 'helm uninstall' was done after the block generation had already stopped.

Thankyou for clearing that up. @antonydenyer any idea what would cause validators to suddenly stop? @Mameta29 do you have logs of the validators at that period we can look at? I'd suggest raising this as an issue on the goquorum repo and closing this - the maintainers there will be able to answer why it suddenly stopped creating blocks.

Thank you for your response. The concern is that when launching a validator, there is a possibility that the keys do not match. For the initial pool (seed validators) the genesis file will create the keys as part of the genesis process and the nodes themselves do not create keys (when you do helm install...). Any validators that get added to the initial set will create their keys and you will need to vote them in (make API calls to the seed validators)

When you do an un install of the validators, what you have in the values.yml will determine wheter the existing keys are deleted or not. We default this to keeping the keys from memory as below

Also, when reviewing the 'values.yml' file, it was configured as follows:

For the four validators: removeKeysOnDelete: false

For the RPC node: removeKeysOnDelete: true

To confirm whether the keys necessary for block generation are mounted correctly within the Kubernetes pods for both validators and the RPC node, you can check the 'keys' directory within each pod. If the keys are correctly mounted, you should be able to see them in the respective 'keys' directories.

Regarding the absence of the genesis file as a pod, it's not typically a pod itself. The genesis file is usually used during the initialization of the network and should be accessible on the file system of the nodes, often outside of the pods.

Not quite sure I follow you here,. The genesis file is created in a pod (as a job) which will exit when done - typically takes less than a couple of mins. It creates the genesis file as a configmap which is used by all nodes

Cheers Josh

Mameta29 commented 11 months ago

@joshuafernandes Thank you for your patience.

@Mameta29 do you have logs of the validators at that period we can look at? I'd suggest raising this as an issue on the goquorum repo and closing this - the maintainers there will be able to answer why it suddenly stopped creating blocks.

The logs are below.

goquorum-node-validator-3-0 goquorum-node-validator-3-quorum TRACE[09-15|03:38:19.517] QBFT: start new ROUND-CHANGE timer       address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" timeout=2560.000
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum INFO [09-15|03:38:19.517] QBFT: start new round                    address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A old.round=7  old.sequence=51045 old.state=Preprepared      old.proposer=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 next.round=8 next.seq=51045 next.proposer=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 next.valSet="[0x2839A248bAA592B8fA4E566e1585493925A2F8A0 0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A 0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7]" next.size=4 next.IsProposer=false
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum WARN [09-15|03:38:19.517] QBFT: TIMER CHANGED ROUND                address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=7 current.sequence=51045 state=Preprepared      pr=<nil>
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum INFO [09-15|03:38:19.518] QBFT: broadcast ROUND-CHANGE message     address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x0000000000000000000000000000000000000000 msg.round=8 msg.sequence=51045 payload=0xf84df849c582c76508c0b841e9e77df4b2df984c845ef587340905219e2e0a38e0d7cbd19c9e5b86b5feaf9d79e3eb9d887cec5a4af008de10782eeb2cdddc8012216f72c05dd02cb564680100c0c0
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum INFO [09-15|03:38:19.517] QBFT: initialize new round               address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=7 current.sequence=51045 target.round=8 lastProposal.number=51044 lastProposal.hash=474faa..cfa230
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum INFO [09-15|03:38:19.517] QBFT: changed state                      address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 old.state=Preprepared      new.state="Accept request"
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.517] QBFT: lookup for pending block proposal requests address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum INFO [09-15|03:38:19.519] QBFT: handle ROUND-CHANGE message        address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=0
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum TRACE[09-15|03:38:19.517] QBFT: start new ROUND-CHANGE timer       address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" timeout=2560.000
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum INFO [09-15|03:38:19.517] QBFT: start new round                    address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 old.round=7  old.sequence=51045 old.state=Preprepared      old.proposer=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 next.round=8 next.seq=51045 next.proposer=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A next.valSet="[0x2839A248bAA592B8fA4E566e1585493925A2F8A0 0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A 0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7]" next.size=4 next.IsProposer=false
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum WARN [09-15|03:38:19.517] QBFT: TIMER CHANGED ROUND                address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=7 current.sequence=51045 state=Preprepared      pr=<nil>
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum INFO [09-15|03:38:19.518] QBFT: broadcast ROUND-CHANGE message     address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x0000000000000000000000000000000000000000 msg.round=8 msg.sequence=51045 payload=0xf84df849c582c76508c0b841662ad0bbea88d83454914543a4ea4eecf59d314376fb18781b6bb2732e4e977f3476ffd93e95c09ad0e435fe6add0c1809dcdb39aac9514593d14b78afff5a1001c0c0
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum INFO [09-15|03:38:19.519] QBFT: handle ROUND-CHANGE message        address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=0
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum TRACE[09-15|03:38:19.519] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum TRACE[09-15|03:38:19.519] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum WARN [09-15|03:38:19.516] QBFT: TIMER CHANGING ROUND               address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=7 current.sequence=51045 state=Preprepared      pr=<nil>
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.519] QBFT: accepted ROUND-CHANGE messages     address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=1
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.524] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.524] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum INFO [09-15|03:38:19.516] QBFT: initialize new round               address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=7 current.sequence=51045 target.round=8 lastProposal.number=51044 lastProposal.hash=5db7a5..307312
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum INFO [09-15|03:38:19.524] QBFT: handle ROUND-CHANGE message        address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=1
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum TRACE[09-15|03:38:19.524] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum TRACE[09-15|03:38:19.524] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.524] QBFT: accepted ROUND-CHANGE messages     address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=2
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.524] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.524] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum INFO [09-15|03:38:19.525] QBFT: handle ROUND-CHANGE message        address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=2
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum TRACE[09-15|03:38:19.525] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum TRACE[09-15|03:38:19.525] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.525] QBFT: accepted ROUND-CHANGE messages     address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum INFO [09-15|03:38:19.526] QBFT: handle ROUND-CHANGE message        address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum TRACE[09-15|03:38:19.526] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum TRACE[09-15|03:38:19.526] QBFT: confirmation Formula used ceil(2N/3) address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.526] QBFT: accepted ROUND-CHANGE messages     address=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=4
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.529] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-1-0 goquorum-node-validator-1-quorum DEBUG[09-15|03:38:19.529] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum INFO [09-15|03:38:19.516] QBFT: changed state                      address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 old.state=Preprepared      new.state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.516] QBFT: lookup for pending block proposal requests address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum TRACE[09-15|03:38:19.519] QBFT: confirmation Formula used ceil(2N/3) address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.516] QBFT: process backlog                    address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 from=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.516] QBFT: start new ROUND-CHANGE timer       address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" timeout=2560.000
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.521] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.521] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [09-15|03:38:19.521] QBFT: handle ROUND-CHANGE message        address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state=Preprepared      msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=4
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum INFO [09-15|03:38:19.516] QBFT: start new round                    address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 old.round=7  old.sequence=51045 old.state=Preprepared      old.proposer=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 next.round=8 next.seq=51045 next.proposer=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 next.valSet="[0x2839A248bAA592B8fA4E566e1585493925A2F8A0 0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A 0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7]" next.size=4 next.IsProposer=false
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum WARN [09-15|03:38:19.516] QBFT: TIMER CHANGED ROUND                address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=7 current.sequence=51045 state=Preprepared      pr=<nil>
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.521] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.521] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state=Preprepared
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum TRACE[09-15|03:38:19.519] QBFT: confirmation Formula used ceil(2N/3) address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.519] QBFT: accepted ROUND-CHANGE messages     address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=1
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.519] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum INFO [09-15|03:38:19.517] QBFT: broadcast ROUND-CHANGE message     address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x0000000000000000000000000000000000000000 msg.round=8 msg.sequence=51045 payload=0xf84df849c582c76508c0b84171aeab2bf4d5b4f31a6c9faf073e11130885fd679339514d58ce75e89c7ebf862d4b9d9115e761f5ef137e43cdbfd37a8bbbe8d2c82aa8110b7234788d32ae0800c0c0
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum INFO [09-15|03:38:19.517] QBFT: handle ROUND-CHANGE message        address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=0
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.517] QBFT: confirmation Formula used ceil(2N/3) address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.517] QBFT: confirmation Formula used ceil(2N/3) address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.517] QBFT: accepted ROUND-CHANGE messages     address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=1
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.519] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum INFO [09-15|03:38:19.519] QBFT: handle ROUND-CHANGE message        address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=1
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum INFO [09-15|03:38:19.520] QBFT: handle ROUND-CHANGE message        address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=1
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum TRACE[09-15|03:38:19.520] QBFT: confirmation Formula used ceil(2N/3) address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum TRACE[09-15|03:38:19.520] QBFT: confirmation Formula used ceil(2N/3) address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.520] QBFT: accepted ROUND-CHANGE messages     address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=2
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum INFO [09-15|03:38:19.520] QBFT: handle ROUND-CHANGE message        address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=2
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum TRACE[09-15|03:38:19.520] QBFT: confirmation Formula used ceil(2N/3) address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum TRACE[09-15|03:38:19.520] QBFT: confirmation Formula used ceil(2N/3) address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.520] QBFT: accepted ROUND-CHANGE messages     address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.521] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.521] QBFT: accepted ROUND-CHANGE messages     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state=Preprepared      msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=8 msg.sequence=51045 higherRoundChanges.count=1 currentRoundChanges.count=4
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.521] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.521] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [09-15|03:38:19.522] QBFT: handle ROUND-CHANGE message        address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state=Preprepared      msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=8 msg.sequence=51045 higherRoundChanges.count=1 currentRoundChanges.count=4
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.522] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [09-15|03:38:19.522] QBFT: received F+1 ROUND-CHANGE messages address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 state=Preprepared      msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=8 msg.sequence=51045 higherRoundChanges.count=2 currentRoundChanges.count=4 F=1
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [09-15|03:38:19.522] QBFT: initialize new round               address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=7 current.sequence=51045 target.round=8 lastProposal.number=51044 lastProposal.hash=474faa..cfa230
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum INFO [09-15|03:38:19.525] QBFT: handle ROUND-CHANGE message        address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.519] QBFT: confirmation Formula used ceil(2N/3) address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum TRACE[09-15|03:38:19.525] QBFT: confirmation Formula used ceil(2N/3) address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.519] QBFT: confirmation Formula used ceil(2N/3) address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum TRACE[09-15|03:38:19.525] QBFT: confirmation Formula used ceil(2N/3) address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.525] QBFT: accepted ROUND-CHANGE messages     address=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=4
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.519] QBFT: accepted ROUND-CHANGE messages     address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=2
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-3-0 goquorum-node-validator-3-quorum DEBUG[09-15|03:38:19.526] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.520] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum INFO [09-15|03:38:19.520] QBFT: handle ROUND-CHANGE message        address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=2
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.520] QBFT: confirmation Formula used ceil(2N/3) address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.520] QBFT: confirmation Formula used ceil(2N/3) address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.520] QBFT: accepted ROUND-CHANGE messages     address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.521] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum INFO [09-15|03:38:19.525] QBFT: handle ROUND-CHANGE message        address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.525] QBFT: confirmation Formula used ceil(2N/3) address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum TRACE[09-15|03:38:19.525] QBFT: confirmation Formula used ceil(2N/3) address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.525] QBFT: accepted ROUND-CHANGE messages     address=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=4
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.526] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.526] consensus message was handled by consensus engine id=04b5aff2e89efc98 conn=inbound    msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[09-15|03:38:19.526] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [09-15|03:38:19.522] QBFT: changed state                      address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 old.state=Preprepared      new.state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.522] QBFT: lookup for pending block proposal requests address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.522] QBFT: process backlog                    address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 from=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.522] QBFT: process backlog                    address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 from=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.522] QBFT: start new ROUND-CHANGE timer       address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request" timeout=2560.000
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [09-15|03:38:19.522] QBFT: start new round                    address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 old.round=7  old.sequence=51045 old.state=Preprepared      old.proposer=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 next.round=8 next.seq=51045 next.proposer=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A next.valSet="[0x2839A248bAA592B8fA4E566e1585493925A2F8A0 0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A 0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7]" next.size=4 next.IsProposer=false
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [09-15|03:38:19.522] QBFT: broadcast ROUND-CHANGE message     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x0000000000000000000000000000000000000000 msg.round=8 msg.sequence=51045 payload=0xf84df849c582c76508c0b8413f65d6308d157b1c40ab244db70de26f0e418cc6e69d1547f6cd8fe172eda7205ba44cb84d505da60c0ee5d04613ef813912d53dae316729b97588ec8c9d619201c0c0
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.523] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [09-15|03:38:19.523] QBFT: handle ROUND-CHANGE message        address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=2
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.523] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.523] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.523] QBFT: accepted ROUND-CHANGE messages     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.524] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [09-15|03:38:19.524] QBFT: handle ROUND-CHANGE message        address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.524] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[09-15|03:38:19.524] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.524] QBFT: accepted ROUND-CHANGE messages     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=8 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=8 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=4
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[09-15|03:38:19.525] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil

It seems to be stuck in a loop of "round change." I wonder that something related that different validator lock the different block?

https://github.com/Consensys/quorum/issues/305

When you do an un install of the validators, what you have in the values.yml will determine wheter the existing keys are deleted or not. We default this to keeping the keys from memory as below

Thank you for this. I checked the genesis.json file. There are correct validator's addresses in the field of "alloc". But at "extraData" It seems to be nothing match of validator's. Please let me make sure that what I should check on the "extraData". After 64 hex characters, There should be account address?

"A list of initial signer addresses (at least one initial signer is required). 20 bytes for each signer."

Signer address mean same as account address like this?

in the validator-3
/keys # cat accountAddress
0x0c25b9b6e66caa1a43a803cb3d5cce4088f07422

Thank you.

joshuafernandes commented 11 months ago

I add some information. I found all validators and a rpc in the /config/static/static-nodes.json

/config/static # cat static-nodes.json
[
  "enode://157c<...some letters...>@goquorum-node-validator-4-0.goquorum-node-validator-4.ryukyu.svc.cluster.local:30303?discport=0",
  "enode://6578<...some letters...>@goquorum-node-validator-1-0.goquorum-node-validator-1.ryukyu.svc.cluster.local:30303?discport=0",
  "enode://c89b<...some letters...>@goquorum-node-rpc-1-0.goquorum-node-rpc-1.ryukyu.svc.cluster.local:30303?discport=0",
  "enode://effe<...some letters...>@goquorum-node-validator-3-0.goquorum-node-validator-3.ryukyu.svc.cluster.local:30303?discport=0",
  "enode://ffb7<...some letters...>@goquorum-node-validator-2-0.goquorum-node-validator-2.ryukyu.svc.cluster.local:30303?discport=0"
]

Yep, this is correct and expected - in goq you can enable discovery and provide bootnodes or a list of static-nodes to connect to

Mameta29 commented 11 months ago

@joshuafernandes

I just add an information. The spec of instance are below.

EC2 type : m5.large
number of type : 7

I found an issue that validator are stopping. Is that something related to this issue?

https://github.com/Consensys/quorum/issues/1630

joshuafernandes commented 11 months ago

Hello @Mameta29

https://github.com/Consensys/quorum/issues/1630 is different in that its potentially a hard fork, and logs clearly have them creating blocks. Your logs show nothing of the sort so blocks aren't being minted.

The genesis file has:

an alloc sections that has account addresses each of which is tied to the validators. These accounts hold ETH to perform txns on the chain
an extraData field which is RLP encoded and contains the node addresses of the seed validators. (See https://github.com/Consensys/quorum-genesis-tool/blob/master/src/lib/nodeKeys.ts#L82C3-L82C3 for how this is done in the QGT) This is an RLP string of all the validators and some padding. So something like 000...V1addressV2addressV3address.... refer to https://besu.hyperledger.org/private-networks/how-to/configure/consensus/clique#two-initial-signers where signer address A/B are the validator addresses

More info on the genesis file here https://consensys.io/blog/hyperledger-besu-how-to-create-an-ethereum-genesis-file.

So I'd suggest you do the following in this order:

Verify that the extraData has the expected validator node addresses in there
Verify that the validator node addresses are being used by each node - to do this check the secret that should have nodekey, nodepub.key and address. When each node comes up the logs should give you an enode - check that the pub key here matches the secrets nodekey.pub
If both the above are as expected, then open a fresh issue in goq and put the logs in there. Based on what you've put in here, this isn't related to infrastructure but more related to something in goq instead and they will be able to sort it out. or
If 1&2 are not as expected then your nodes aren't seeing the nodekeys and are creating a new key each time which would also cause the behaviour you're seeing. To fix this give them the correct keys and restart all validators. If you are seeing this, something has changed externally on your cluster that has caused this ie the secrets have been altered and the nodes have created new ones . You'd need to check the k8s event log to see what changed, maybe a helm uninstall or similar, but this is not behavior that is in the charts themselves. Restarts persist keys on the repo https://github.com/Consensys/quorum-kubernetes/blob/master/helm/charts/goquorum-node/templates/node-hooks-pre-delete.yaml#L122 has this defaulted to false for this reason

Mameta29 commented 11 months ago

@joshuafernandes Thank you for your kind response. I will try and check those you suggested me.

Could you check if the fork happened with the logs below. The validator know the same number of block heigh but different block hash. https://github.com/Consensys/quorum-kubernetes/issues/207#issuecomment-1744207518

goquorum-node-validator-2-0 goquorum-node-validator-2-quorum DEBUG[10-03|04:40:44.103] Current full block not old enough        number=51044 hash=5db7a5..307312 delay=3,162,240
.
.
.
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-03|04:40:46.560] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240

Mameta29 commented 11 months ago

@joshuafernandes

Regarding the method to verify the extraData and the pub key as mentioned above, could you please confirm if my approach is correct? Also, if my verification method seems accurate, what should be the next steps based on these results? Does a discrepancy in extraData imply that the validator addresses have changed, making it impossible to restore this chain?

extraData

Verify that the extraData has the expected validator node addresses in there

genesis.json

/etc/genesis # cat genesis.json
{
"nonce": "0x0",
"timestamp": "0x58ee40ba",
"extraData": "0xf87aa00000000000000000000000000000000000000000000000000000000000000000f85494ad31e0734ee313eb2e708d2bc538fb8c332f60e4942839a248baa592b8fa4e566e1585493925a2f8a094c23f4cf6b1dbc6323c1eb9fb749a556e5a43db5a94c62ecb2c35a25dd71bd1c92a6cbccecc5698b2a7c080c0",
"gasLimit": "0x47b760",
.
.
.
},
"alloc": {
"0x111f0f5e627b332ef26a2e7c507204a85ac52bbb": {
  "balance": "1000000000000000000000000000"
},
"0xa874f9df79de99f3e558f1cdba7f7c6af33a5434": {
  "balance": "1000000000000000000000000000"
},
"0x0c25b9b6e66caa1a43a803cb3d5cce4088f07422": {
  "balance": "1000000000000000000000000000"
},
"0x09d0f31105244da2a79af132d32b8dcf11c2e9b8": {
  "balance": "1000000000000000000000000000"
}
}

Code of get extraData


import RLP, { Input } from 'rlp';
export type Address = Buffer;

const validatorAddresses = [ "0x111f0f5e627b332ef26a2e7c507204a85ac52bbb", "0xa874f9df79de99f3e558f1cdba7f7c6af33a5434", "0x0c25b9b6e66caa1a43a803cb3d5cce4088f07422", "0x09d0f31105244da2a79af132d32b8dcf11c2e9b8" ];

const validatorAddressBuffers: Address[] = validatorAddresses.map(address => Buffer.from(address.slice(2), 'hex')); function generateExtraDataString(validatorAddressList: Address[]): string { let extraData: Input; const extraDataContent: Input = [new Uint8Array(32), validatorAddressList, [], null, []]; const extraDataCoded: Uint8Array = RLP.encode(extraDataContent); extraData = '0x' + Buffer.from(extraDataCoded).toString('hex'); return extraData; }

const extraData = generateExtraDataString(validatorAddressBuffers); console.log("Generated extraData:", extraData);


- the result of extraData

Generated extraData: 0xf87aa00000000000000000000000000000000000000000000000000000000000000000f85494111f0f5e627b332ef26a2e7c507204a85ac52bbb94a874f9df79de99f3e558f1cdba7f7c6af33a5434940c25b9b6e66caa1a43a803cb3d5cce4088f074229409d0f31105244da2a79af132d32b8dcf11c2e9b8c080c0


2. Pub key

> Verify that the validator node addresses are being used by each node - to do this check the secret that should have nodekey, nodepub.key and address. When each node comes up the logs should give you an enode - check that the pub key here matches the secrets nodekey.pub

/config/static # cat static-nodes.json [ "enode://157c84a6a886d251760e64ecf558407148370250d81eae65571921646040f861cf3ee3867f3a8be0aefc9a1c25b5c6378e402051932923e323d9bd824769282f@goquorum-node-validator-4-0.goquorum-node-validator-4.ryukyu.svc.cluster.local:30303?discport=0", "enode://6578ba89b29c4ee4789bee6a6a3b78165bf29013f7d9c3ce01362fcec1b96a0eb6827c3f69fd1998eabec22f2c6ba209b6c745afe49255df9927912d6e7fba36@goquorum-node-validator-1-0.goquorum-node-validator-1.ryukyu.svc.cluster.local:30303?discport=0", "enode://c89bae5a69e8e626fde22e300cd98a8f2281b40340a9e101308fa250dc0e577f2c7c2f6bfe4883a3c51721ef57347cf910b95db75c94cb67c01065c81b0e5e1b@goquorum-node-rpc-1-0.goquorum-node-rpc-1.ryukyu.svc.cluster.local:30303?discport=0", "enode://effec4a33ddfe157381fe19fdc1e1ef3c6c29d9c1adce157c9579b7160eed21f97a8645eb1c7eb64c0909dfdca7eceb3446dc368ae07664b3e3466f5ff7f74d4@goquorum-node-validator-3-0.goquorum-node-validator-3.ryukyu.svc.cluster.local:30303?discport=0", "enode://ffb765a3fdc8e0d01480cf3de91327eaf9bb8652dc4edfb854a376525b0e2d342473bef5650ce73c886a0eace144b8a73202e1e92a48167080e8cd4053a77ffc@goquorum-node-validator-2-0.goquorum-node-validator-2.ryukyu.svc.cluster.local:30303?discport=0" ]


- goquorum-node-validator-1

/keys # cat accountAddress 0x111f0f5e627b332ef26a2e7c507204a85ac52bbb/keys #

/keys # cat nodekey.pub 6578ba89b29c4ee4789bee6a6a3b78165bf29013f7d9c3ce01362fcec1b96a0eb6827c3f69fd1998eabec22f2c6ba209b6c745afe49255df9927912d6e7fba36/keys #

/keys # cat enode 6578ba89b29c4ee4789bee6a6a3b78165bf29013f7d9c3ce01362fcec1b96a0eb6827c3f69fd1998eabec22f2c6ba209b6c745afe49255df9927912d6e7fba36/keys #



The other three validators also had the same values for enode and nodekey.pub. While the extraData mentioned in point 1 is different, am I correct in understanding that the Pub key mentioned in point 2 is accurate?

Mameta29 commented 11 months ago

@joshuafernandes

I've just updated to have found the "error logs" below. QBFT: Error List() Signed Payload err="rlp: end of list"

goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:07:48.667] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:07:53.325] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:08:13.326] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:08:33.327] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:08:48.668] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:08:53.327] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:09:13.328] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:09:33.328] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:09:48.668] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:09:53.329] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:10:13.330] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:10:33.331] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:10:48.668] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:10:53.331] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:01.879] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:01.879] QBFT: handle ROUND-CHANGE message        address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=14 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=15 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=2
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:01.879] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=14 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:01.879] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=14 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:01.879] QBFT: accepted ROUND-CHANGE messages     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=14 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=15 msg.sequence=51045 higherRoundChanges.count=1 currentRoundChanges.count=2
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:01.879] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:01.880] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.128] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.128] QBFT: handle ROUND-CHANGE message        address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=14 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=15 msg.sequence=51045 higherRoundChanges.count=1 currentRoundChanges.count=2
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.128] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=14 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.128] QBFT: received F+1 ROUND-CHANGE messages address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=14 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=15 msg.sequence=51045 higherRoundChanges.count=2 currentRoundChanges.count=2 F=1
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.128] QBFT: initialize new round               address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=14 current.sequence=51045 target.round=15 lastProposal.number=51044 lastProposal.hash=474faa..cfa230
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.128] QBFT: lookup for pending block proposal requests address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.128] QBFT: start new ROUND-CHANGE timer       address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request" timeout=327680.000
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.128] QBFT: start new round                    address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 old.round=14 old.sequence=51045 old.state="Accept request" old.proposer=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 next.round=15 next.seq=51045 next.proposer=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 next.valSet="[0x2839A248bAA592B8fA4E566e1585493925A2F8A0 0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A 0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7]" next.size=4 next.IsProposer=false
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.129] QBFT: broadcast ROUND-CHANGE message     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x0000000000000000000000000000000000000000 msg.round=15 msg.sequence=51045 payload=0xf84df849c582c7650fc0b841bd85ebfe6345484dc71d990dab7e9184396915ea8ed8ea81b76c732f93bce8195088abc5dbfdbc378cf5e04e5e0ea36e13b5ad3d754c962a15e09f3467aee27800c0c0
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.129] QBFT: handle ROUND-CHANGE message        address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=15 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=2
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.129] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.129] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.129] QBFT: accepted ROUND-CHANGE messages     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=15 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.130] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.130] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.130] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.131] QBFT: handle ROUND-CHANGE message        address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=15 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.131] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.131] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.131] QBFT: accepted ROUND-CHANGE messages     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request" msg.code=21 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=15 msg.sequence=51045 higherRoundChanges.count=0 currentRoundChanges.count=4
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.132] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=staticdial msg=21 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.132] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=18 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.132] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=51045, round=15, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.132] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=51045, round=15, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.132] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=51045, round=15, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum ERROR[10-05|02:11:02.132] QBFT: Error List() Signed Payload        err="rlp: end of list"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.133] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=18 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.133] QBFT: handle PRE-PREPARE message         address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum WARN [10-05|02:11:02.133] QBFT: ignore PRE-PREPARE message from non proposer address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request" msg.code=18 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=15 msg.sequence=51045 proposal.number=51045 proposal.hash=0x3aa2940062ff49b05be7d729ae66723d3842545fb987e1bcf0547ee818d19bab proposer=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.133] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=51045, round=15, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.133] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=51045, round=15, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.133] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=51045, round=15, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum ERROR[10-05|02:11:02.133] QBFT: Error List() Signed Payload        err="rlp: end of list"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.134] QBFT: handle PRE-PREPARE message         address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.134] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.134] QBFT: hasQuorumOfRoundChangeMessagesForNil rc="RoundChange {seq=51045, round=15, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.134] QBFT: hasQuorumOfRoundChangeMessagesForNil rc="RoundChange {seq=51045, round=15, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.134] QBFT: hasQuorumOfRoundChangeMessagesForNil rc="RoundChange {seq=51045, round=15, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.134] BFT: loaded voting snapshot from cache   snap.number=51044 snap.hash=0x474faa110376241205ee9da2f1fe10681ccdd47116e54fc1dc0bd4eeb5cfa230 snap.epoch=30000 snap.validators="[0x2839A248bAA592B8fA4E566e1585493925A2F8A0 0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A 0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7]" snap.votes=[]
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.134] QBFT: accepted PRE-PREPARE message       address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.134] QBFT: start new ROUND-CHANGE timer       address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state="Accept request" timeout=327680.000
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.134] QBFT: changed state                      address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 old.state="Accept request" new.state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.134] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=19 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.134] QBFT: broadcast PREPARE message          address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared      msg.code=19 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=15 msg.sequence=51045 payload=0xf869e582c7650fa0ac484576229acf53e4a75469c66cd5f23734077289f2ce37b722cda416caf1f3b841e7e7d0a60c1d24bc16c683b2675e7ebe13120f6d4248cec7c8f6525f6949883014cf8e89ff6d0c1904bda791f4dfef4657a43f011f5e67381d9b6e30416a04b401
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.134] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=staticdial msg=19 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.135] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=staticdial msg=18 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.135] consensus message was handled by consensus engine id=51f3b1c13847e9bb conn=staticdial msg=19 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.135] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.135] QBFT: handle PREPARE message             address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared      msg.code=19 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=15 msg.sequence=51045 prepares.count=0 quorum=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum ERROR[10-05|02:11:02.135] QBFT: invalid PREPARE message digest     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared      msg.code=19 msg.source=0xc23F4Cf6b1dbc6323c1EB9fB749a556e5a43db5A msg.round=15 msg.sequence=51045
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.135] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.135] QBFT: handle PREPARE message             address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared      msg.code=19 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=15 msg.sequence=51045 prepares.count=0 quorum=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum ERROR[10-05|02:11:02.135] QBFT: invalid PREPARE message digest     address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared      msg.code=19 msg.source=0x2839A248bAA592B8fA4E566e1585493925A2F8A0 msg.round=15 msg.sequence=51045
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.136] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.136] QBFT: handle PREPARE message             address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared      msg.code=19 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=15 msg.sequence=51045 prepares.count=0 quorum=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.136] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.136] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.136] QBFT: accepted PREPARE messages          address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared      msg.code=19 msg.source=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 msg.round=15 msg.sequence=51045 prepares.count=1 quorum=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.136] consensus message was handled by consensus engine id=d3279e9f5fc0d6e0 conn=staticdial msg=19 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.139] consensus message was handled by consensus engine id=e941cb8a01018f6c conn=staticdial msg=19 quorumConsensusProtocolName=istanbul err=nil
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.139] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum INFO [10-05|02:11:02.139] QBFT: handle PREPARE message             address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared      msg.code=19 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=15 msg.sequence=51045 prepares.count=1 quorum=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.139] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum TRACE[10-05|02:11:02.139] QBFT: confirmation Formula used ceil(2N/3) address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:02.139] QBFT: accepted PREPARE messages          address=0xc62ecB2C35a25DD71Bd1C92a6cbcCECc5698B2A7 current.round=15 current.sequence=51045 state=Preprepared      msg.code=19 msg.source=0xAd31e0734eE313eB2e708D2bC538fb8c332f60E4 msg.round=15 msg.sequence=51045 prepares.count=2 quorum=3
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:13.332] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:33.332] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:48.668] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:11:53.333] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:12:13.333] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:12:33.333] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:12:48.669] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:12:53.333] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:13:13.334] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:13:33.334] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:13:48.670] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:13:53.335] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:14:13.335] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:14:33.335] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:14:48.670] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:14:53.335] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:15:13.336] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:15:33.337] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:15:48.671] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:15:53.337] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:16:13.338] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:16:33.339] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
goquorum-node-validator-4-0 goquorum-node-validator-4-quorum DEBUG[10-05|02:16:48.673] Current full block not old enough        number=51044 hash=474faa..cfa230 delay=3,162,240

joshuafernandes commented 11 months ago

Hello @Mameta29

Right I'll give some more context:

GoQ uses 2 types of addresses:

The nodekey that identifies this node as unique on the network. Think of this as your drivers license - its unique to you and you use to verify your identity. The node's pubkey & address that can be derived from the nodekey. So for example in the playground section we have the following secret for each validator https://github.com/Consensys/quorum-kubernetes/blob/master/playground/kubectl/quorum-go/ibft/secrets/validator1-keys-secret.yaml#L10 and we only care about line 10 for the moment. This nodekey is then save to the data dir /data/geth/nodekey
The second is the account key which is tied to the wallet - typically this is encrypted (and you provide the password to use it) and is used as gas (ETH) for transactions. Think of this one as your credit card - you use this to buy/sell/transact and not your driver's license. So continuing with the same file in the playground we now use line 12 & 14 https://github.com/Consensys/quorum-kubernetes/blob/master/playground/kubectl/quorum-go/ibft/secrets/validator1-keys-secret.yaml#L12 This one is saved to /data/keystore/key and the address here is used in the alloc section - ie its this address that is used to transact and so you fund the account with a given amount

For what you are doing with verifying which validators are in the extradata we only care about item 1. We don;t need the alloc at all. So getting back to the nodekey:

once it is in /data/geth/nodekey its corresponding pubkey is derived and that is what you see as the enode. So in the playground for validator 1 its nodekey=1a2c4ff0f1b38e2322658dba692816138eb22d002515df1fffca21278f406aa9 and its corresponding pubkey is 8208a3f344695d44e9cf2c023683cbea7b9343e2f70a5e804bd2c93858e945f8f91439eef96a4ab6c47ff06637d6fbe6472f96de1655a1bee57ea896654f3a22 so the logs/static-nodes/etc refer to it as enode://8208a3f344695d44e9cf2c023683cbea7b9343e2f70a5e804bd2c93858e945f8f91439eef96a4ab6c47ff06637d6fbe6472f96de1655a1bee57ea896654f3a22@validator1-0.quorum-validator1.quorum.svc.cluster.local:30303?discport=0. Its corresponding address is 0x93917cadbace5dfce132b991732c6cda9bcc5b8a This is the address we care about it for the extraData in the genesis.

You can also see all the files (for 1 & 2 that we use in the quickstart https://github.com/Consensys/quorum-dev-quickstart/tree/master/files/common/config/nodes/validator1)

In the genesis file we then have the extra data made up of 4 address and padding (1 of those address is the one above) "extraData": "0x0000000000000000000000000000000000000000000000000000000000000000f89af8549493917cadbace5dfce132b991732c6cda9bcc5b8a9427a97c9aaf04f18f3014c32e036dd0ac76da5f1894ce412f988377e31f4d0ff12d74df73b51c42d0ca9498c1334496614aed49d2e81526d089f7264fed9cb8410000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0", which when broken down is


94
93917cadbace5dfce132b991732c6cda9bcc5b8a    # v1 address
94
27a97c9aaf04f18f3014c32e036dd0ac76da5f18    # v2 address
94
ce412f988377e31f4d0ff12d74df73b51c42d0ca    # v3 address
94
98c1334496614aed49d2e81526d089f7264fed9c    # v4 address
b841
0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0 # more padding```

Do not mix alloc accounts with validator addresses - they are both called the same but are different ie you wouldnt buy anything with your drivers license but only your credit card and you can't really use your credit card as ID like your drivers license. Hope this helps. If it makes ie easier use the quickstart (it has the same keys and addresses) so you can see whats happening. In your case you need to verify that in the genesis the nodekey addresses match up.

joshuafernandes commented 11 months ago

The goq team should be able to help figure out what happened with the nodes, looks like their either not peered and each creating their own block proposal or something like that.

Consensys / quorum-kubernetes

The Round Change continues indefinitely, BUT no new block is being generated. #207

Overview: