search

Consensys / quorum-kubernetes

Helm charts for Hyperledger Besu and GoQuorum
Apache License 2.0
119 stars 128 forks source link

Unable to find sync target in a bootnode #239

Closed eldon-tcu closed 2 months ago

eldon-tcu commented 2 months ago

I have been trying to setup an environment Hyperledger Besu on WSL with microk8s. I did follow the instructions and successfully run and verify the following steps:

After the deploy of bootnodes, I got the good message:

> microk8s helm3 install bootnode-1 ./charts/besu-node --namespace besu --values ./values/bootnode.yml
NAME: bootnode-1
LAST DEPLOYED: Thu Aug 15 19:34:04 2024
NAMESPACE: besu
STATUS: deployed
REVISION: 1
TEST SUITE: None

Thus, I run the second bootnode and four validators:

quorum-kubernetes/helm [ master][!]
> microk8s helm3 install bootnode-2 ./charts/besu-node --namespace besu --values ./values/bootnode.yml
NAME: bootnode-2
LAST DEPLOYED: Thu Aug 15 20:30:40 2024
NAMESPACE: besu
STATUS: deployed
REVISION: 1
TEST SUITE: None

quorum-kubernetes/helm [ master][!][⏱ 8s]
> microk8s helm3 install validator-1 ./charts/besu-node --namespace besu --values ./values/validator.yml
NAME: validator-1
LAST DEPLOYED: Thu Aug 15 20:35:19 2024
NAMESPACE: besu
STATUS: deployed
REVISION: 1
TEST SUITE: None

quorum-kubernetes/helm [ master][!][⏱ 7s]
> microk8s helm3 install validator-2 ./charts/besu-node --namespace besu --values ./values/validator.yml
NAME: validator-2
LAST DEPLOYED: Thu Aug 15 20:35:55 2024
NAMESPACE: besu
STATUS: deployed
REVISION: 1
TEST SUITE: None

quorum-kubernetes/helm [ master][!][⏱ 8s]
> microk8s helm3 install validator-3 ./charts/besu-node --namespace besu --values ./values/validator.yml
NAME: validator-3
LAST DEPLOYED: Thu Aug 15 20:36:23 2024
NAMESPACE: besu
STATUS: deployed
REVISION: 1
TEST SUITE: None

quorum-kubernetes/helm [ master][!][⏱ 8s]
> microk8s helm3 install validator-4 ./charts/besu-node --namespace besu --values ./values/validator.yml
NAME: validator-4
LAST DEPLOYED: Thu Aug 15 20:36:44 2024
NAMESPACE: besu
STATUS: deployed
REVISION: 1
TEST SUITE: None

quorum-kubernetes/helm [ master][!][⏱ 8s]

Then, we verified the results on Kubernetes:

NAME                                                                      READY STATUS                 RESTARTS       AGE
pod/besu-genesis-init-sm4ns                                  0/1     Completed              0              68m
pod/besu-node-bootnode-1-0                                   1/1     Running                0              65m
pod/besu-node-bootnode-2-0                                   1/1     Running                0              8m25s
pod/besu-node-validator-1-0                                  1/1     Running                0              3m47s
pod/besu-node-validator-2-0                                  1/1     Running                0              3m10s
pod/besu-node-validator-3-0                                  1/1     Running                0              2m42s
pod/besu-node-validator-4-0                                  1/1     Running                0              2m21s

NAME                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                   AGE
service/alertmanager-operated                     ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP                                112m
service/besu-node-bootnode-1                      ClusterIP   10.152.183.73    <none>        8545/TCP,8546/TCP,8547/TCP,30303/TCP,30303/UDP,9545/TCP   65m
service/besu-node-bootnode-2                      ClusterIP   10.152.183.65    <none>        8545/TCP,8546/TCP,8547/TCP,30303/TCP,30303/UDP,9545/TCP   8m25s
service/besu-node-validator-1                     ClusterIP   10.152.183.195   <none>        8545/TCP,8546/TCP,8547/TCP,30303/TCP,30303/UDP,9545/TCP   3m47s
service/besu-node-validator-2                     ClusterIP   10.152.183.85    <none>        8545/TCP,8546/TCP,8547/TCP,30303/TCP,30303/UDP,9545/TCP   3m10s
service/besu-node-validator-3                     ClusterIP   10.152.183.84    <none>        8545/TCP,8546/TCP,8547/TCP,30303/TCP,30303/UDP,9545/TCP   2m42s
service/besu-node-validator-4                     ClusterIP   10.152.183.212   <none>        8545/TCP,8546/TCP,8547/TCP,30303/TCP,30303/UDP,9545/TCP   2m21s
service/monitoring-grafana                        ClusterIP   10.152.183.186   <none>        80/TCP                                                    113m
service/monitoring-kube-prometheus-alertmanager   ClusterIP   10.152.183.108   <none>        9093/TCP                                                  113m
service/monitoring-kube-prometheus-operator       ClusterIP   10.152.183.71    <none>        443/TCP                                                   113m
service/monitoring-kube-prometheus-prometheus     ClusterIP   10.152.183.36    <none>        9090/TCP                                                  113m
service/monitoring-kube-state-metrics             ClusterIP   10.152.183.45    <none>        8080/TCP                                                  113m
service/monitoring-prometheus-node-exporter       ClusterIP   10.152.183.135   <none>        9100/TCP                                                  113m
service/prometheus-operated                       ClusterIP   None             <none>        9090/TCP                                                  112m

But, when I inspect the logs on pod with:

kubectl logs besu-node-bootnode-1-0 -n besu

I got:

kubectl logs besu-node-bootnode-1-0 -n besu
Defaulted container "bootnode-1-besu" out of: bootnode-1-besu, volume-permission-besu (init)
Setting logging level to INFO
2024-08-15 22:34:36.530+00:00 | main | INFO  | Besu | Starting Besu
2024-08-15 22:34:37.031+00:00 | main | WARN  | Besu | --graphql-http-host has been ignored because --graphql-http-enabled was not defined on the command line.
2024-08-15 22:34:37.508+00:00 | main | WARN  | Besu | Permissions are disabled. Cannot enable PERM APIs when not using Permissions.
2024-08-15 22:34:37.601+00:00 | main | INFO  | Besu | Connecting to 5 static nodes.
2024-08-15 22:34:37.607+00:00 | main | INFO  | Besu | Forcing price bump for transaction replacement to 0, since we are on a zero basefee network
2024-08-15 22:34:37.608+00:00 | main | INFO  | Besu | Forcing price bump for transaction replacement to 0, since min-gas-price is set to 0
2024-08-15 22:34:37.609+00:00 | main | WARN  | Besu | Forcing tx-pool-min-gas-price=0, since it cannot be greater than the value of min-gas-price
2024-08-15 22:34:37.804+00:00 | main | INFO  | Besu |
####################################################################################################
#                                                                                                  #
# Besu version 24.3.3                                                                              #
#                                                                                                  #
# Configuration:                                                                                   #
# Network: Custom genesis file                                                                     #
# /etc/genesis/genesis.json                                                                        #
# Network Id: 1337                                                                                 #
# Data storage: Bonsai                                                                             #
# Sync mode: Full                                                                                  #
# RPC HTTP APIs: DEBUG,ETH,ADMIN,WEB3,IBFT,NET,TRACE,EEA,PRIV,QBFT,PERM,TXPOOL                     #
# RPC HTTP port: 8545                                                                              #
# Using LAYERED transaction pool implementation                                                    #
# Using STACKED worldstate update mode                                                             #
#                                                                                                  #
# Host:                                                                                            #
# Java: openjdk-java-17                                                                            #
# Maximum heap size: 0.45 GB                                                                       #
# OS: linux-x86_64                                                                                 #
# glibc: 2.35                                                                                      #
# jemalloc: 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756                                      #
# Total memory: 15.62 GB                                                                           #
# CPU cores: 4                                                                                     #
#                                                                                                  #
# Plugins:                                                                                         #
# TOTAL = 0 of 0 plugins successfully loaded                                                       #
# from /opt/besu/plugins                                                                           #
#                                                                                                  #
####################################################################################################
2024-08-15 22:34:37.806+00:00 | main | INFO  | Besu | Security Module: localfile
2024-08-15 22:34:37.807+00:00 | main | INFO  | VersionMetadata | Lookup version metadata file in data directory: /data/besu
2024-08-15 22:34:37.807+00:00 | main | INFO  | VersionMetadata | No version data detected. Writing Besu version 24.3.3 to metadata file
2024-08-15 22:34:37.928+00:00 | main | INFO  | Besu | Using the native implementation of alt bn128
2024-08-15 22:34:38.009+00:00 | main | INFO  | Besu | Using the native implementation of modexp
2024-08-15 22:34:38.009+00:00 | main | INFO  | Besu | Using the native implementation of the signature algorithm
2024-08-15 22:34:38.012+00:00 | main | INFO  | Besu | Using the native implementation of the blake2bf algorithm
2024-08-15 22:34:38.325+00:00 | main | INFO  | Besu | Forcing price bump for transaction replacement to 0, since we are on a zero basefee network
2024-08-15 22:34:38.326+00:00 | main | INFO  | Besu | Forcing price bump for transaction replacement to 0, since min-gas-price is set to 0
2024-08-15 22:34:38.327+00:00 | main | WARN  | Besu | Forcing tx-pool-min-gas-price=0, since it cannot be greater than the value of min-gas-price
2024-08-15 22:34:38.330+00:00 | main | INFO  | KeyPairUtil | Attempting to load public key from /keys/nodekey
2024-08-15 22:34:38.393+00:00 | main | INFO  | KeyPairUtil | Loaded public key 0x305e51bb6e310b178ab072e1e6f89d7b2273415ddbff8ffc98397286eee315be49bd87e4085a34c82a320da6f268824c90ada37b6abd0d5aa34fa0a9380fc882 from /keys/nodekey
2024-08-15 22:34:38.398+00:00 | main | WARN  | Besu | Privacy is disabled. Cannot use EEA/PRIV API methods when not using Privacy.
2024-08-15 22:34:38.711+00:00 | main | INFO  | ProtocolScheduleBuilder | Protocol schedule created with milestones: [London:0]
2024-08-15 22:34:38.727+00:00 | main | INFO  | FlatDbStrategyProvider | Bonsai flat db mode found PARTIAL
2024-08-15 22:34:38.729+00:00 | main | INFO  | FlatDbStrategyProvider | Bonsai db mode with code stored using code hash enabled = false
2024-08-15 22:34:38.820+00:00 | main | INFO  | RocksDBKeyValueStorageFactory | No existing database detected at /data/besu. Using version 2
2024-08-15 22:34:39.303+00:00 | main | INFO  | FlatDbStrategyProvider | Bonsai flat db mode found PARTIAL
2024-08-15 22:34:39.305+00:00 | main | INFO  | FlatDbStrategyProvider | Bonsai db mode with code stored using code hash enabled = false
2024-08-15 22:34:39.506+00:00 | main | INFO  | TransactionPoolFactory | Enabling transaction pool
2024-08-15 22:34:39.529+00:00 | main | INFO  | BesuControllerBuilder | TTD difficulty is not present, creating initial sync phase for PoW
2024-08-15 22:34:39.620+00:00 | main | INFO  | QbftBesuControllerBuilder | Starting QBFT mining coordinator
2024-08-15 22:34:39.629+00:00 | main | INFO  | RunnerBuilder | Resolved 0 bootnodes.
2024-08-15 22:34:39.632+00:00 | main | INFO  | RunnerBuilder | Detecting NAT service.
2024-08-15 22:34:39.914+00:00 | main | INFO  | Runner | Starting external services ...
2024-08-15 22:34:39.915+00:00 | main | INFO  | MetricsHttpService | Starting metrics http service on 0.0.0.0:9545
2024-08-15 22:34:40.120+00:00 | main | INFO  | JsonRpcHttpService | Starting JSON-RPC service on 0.0.0.0:8545
2024-08-15 22:34:40.119+00:00 | vert.x-eventloop-thread-1 | INFO  | MetricsHttpService | Metrics service started and listening on 0.0.0.0:9545
2024-08-15 22:34:40.130+00:00 | vert.x-eventloop-thread-1 | INFO  | JsonRpcHttpService | JSON-RPC service started and listening on 0.0.0.0:8545
2024-08-15 22:34:40.191+00:00 | main | INFO  | WebSocketService | Starting Websocket service on 0.0.0.0:8546
2024-08-15 22:34:40.194+00:00 | vert.x-eventloop-thread-1 | INFO  | WebSocketService | Websocket service started and listening on 0.0.0.0:8546
2024-08-15 22:34:40.196+00:00 | main | INFO  | AutoTransactionLogBloomCachingService | Starting auto transaction log bloom caching service.
2024-08-15 22:34:40.198+00:00 | main | INFO  | LogBloomCacheMetadata | Lookup cache metadata file in data directory: /data/besu/caches
2024-08-15 22:34:40.208+00:00 | main | INFO  | Runner | Starting Ethereum main loop ...
2024-08-15 22:34:40.208+00:00 | main | INFO  | KubernetesNatManager | Starting kubernetes NAT manager.
2024-08-15 22:34:41.128+00:00 | main | INFO  | KubernetesNatManager | Found Besu service: besu-node-bootnode-1
2024-08-15 22:34:41.129+00:00 | main | INFO  | KubernetesNatManager | Setting host IP to: 10.152.183.73.
2024-08-15 22:34:41.192+00:00 | main | WARN  | KubernetesNatManager | Ignored unknown Besu port: Not an integer
2024-08-15 22:34:41.193+00:00 | main | WARN  | KubernetesNatManager | Ignored unknown Besu port: Invalid NAT service type provided: ws
2024-08-15 22:34:41.193+00:00 | main | WARN  | KubernetesNatManager | Ignored unknown Besu port: Invalid NAT service type provided: graphql
2024-08-15 22:34:41.193+00:00 | main | WARN  | KubernetesNatManager | Ignored unknown Besu port: Not an integer
2024-08-15 22:34:41.194+00:00 | main | WARN  | KubernetesNatManager | Ignored unknown Besu port: Not an integer
2024-08-15 22:34:41.194+00:00 | main | WARN  | KubernetesNatManager | Ignored unknown Besu port: Invalid NAT service type provided: metrics
2024-08-15 22:34:41.194+00:00 | main | INFO  | NetworkRunner | Starting Network.
2024-08-15 22:34:41.208+00:00 | nioEventLoopGroup-2-1 | INFO  | RlpxAgent | P2P RLPx agent started and listening on /[0:0:0:0:0:0:0:0]:30303.
2024-08-15 22:34:41.208+00:00 | main | INFO  | PeerDiscoveryAgent | Starting peer discovery agent on host=0.0.0.0, port=30303
2024-08-15 22:34:41.224+00:00 | vert.x-eventloop-thread-1 | INFO  | VertxPeerDiscoveryAgent | Started peer discovery agent successfully, on effective host=0:0:0:0:0:0:0:0%0 and port=30303
2024-08-15 22:34:41.291+00:00 | vert.x-eventloop-thread-1 | INFO  | PeerDiscoveryAgent | P2P peer discovery agent started and listening on /[0:0:0:0:0:0:0:0%0]:30303
2024-08-15 22:34:41.391+00:00 | vert.x-eventloop-thread-1 | INFO  | PeerDiscoveryAgent | Writing node record to disk. NodeRecord{seq=1, publicKey=0x02305e51bb6e310b178ab072e1e6f89d7b2273415ddbff8ffc98397286eee315be, udpAddress=Optional[/10.152.183.73:30303], tcpAddress=Optional[/10.152.183.73:30303], asBase64=-Je4QHjQuO7TTeB7qHa_TeLU8VMfC1tLz5bTUxSvQswfHe8zMWFEmu4GFpu_xZUMskd3uloB6_lKf_ywwLxVx9kvgRcBg2V0aMfGhKVWvCmAgmlkgnY0gmlwhAqYt0mJc2VjcDI1NmsxoQIwXlG7bjELF4qwcuHm-J17InNBXdv_j_yYOXKG7uMVvoN0Y3CCdl-DdWRwgnZf, nodeId=0xfaf558bb6fef8a1b162d2cab73f4b68afc2a3b7aa0940fbd958cccbd74b4df45, customFields={tcp=30303, udp=30303, ip=0x0a98b749, eth=[[0xa556bc29, 0x]], id=V4, secp256k1=0x02305e51bb6e310b178ab072e1e6f89d7b2273415ddbff8ffc98397286eee315be}}
2024-08-15 22:34:41.404+00:00 | main | INFO  | DefaultP2PNetwork | Enode URL enode://305e51bb6e310b178ab072e1e6f89d7b2273415ddbff8ffc98397286eee315be49bd87e4085a34c82a320da6f268824c90ada37b6abd0d5aa34fa0a9380fc882@10.152.183.73:30303
2024-08-15 22:34:41.405+00:00 | main | INFO  | DefaultP2PNetwork | Node address 0x73f4b68afc2a3b7aa0940fbd958cccbd74b4df45
2024-08-15 22:34:41.425+00:00 | main | INFO  | NetworkRunner | Supported capabilities: [eth/62, eth/63, eth/64, eth/65, eth/66, eth/67, eth/68], [istanbul/100], [snap/1]
2024-08-15 22:34:41.425+00:00 | main | INFO  | DefaultSynchronizer | Starting synchronizer.
2024-08-15 22:34:41.426+00:00 | main | INFO  | TransactionPoolFactory | Enabling transaction handling following initial sync
2024-08-15 22:34:41.427+00:00 | main | INFO  | QbftBesuControllerBuilder | Starting QBFT mining coordinator following initial sync
2024-08-15 22:34:41.428+00:00 | main | INFO  | FullSyncDownloader | Starting full sync.
2024-08-15 22:34:41.429+00:00 | main | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:34:41.500+00:00 | main | INFO  | Runner | Ethereum main loop is up.
2024-08-15 22:34:46.502+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:34:51.503+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:34:56.504+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:35:01.505+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:35:06.507+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:35:11.509+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:35:16.510+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:35:21.511+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:35:26.512+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:35:31.513+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness
2024-08-15 22:35:36.514+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | Unable to find sync target. Currently checking 0 peers for usefulness

I appreciate some help to solve the sync problem. I thank you in advance

eldon-tcu commented 2 months ago

I just restart my environment (microk8s) and follow the "script" again and it works fine now. I didn't know what was wrong with the first attempt. Sorry for the inconvenience of the previous question. I haven't published the RPC and Transaction yet and I'm still having some monitoring problems. I will work on it and if necessary I will ask for your help again. Cheers

joshuafernandes commented 2 months ago

Hello @eldon-tcu sounds good, That message from the bootnodes is normal. When the validators start there are rounds that each validator takes to create a block - when they start producing blocks that will get transmitted to the other rpc + bootnodes.

I'll close this for now but feel free to give us a yell if you run into issues.

Cheers