midl-dev / tezos-on-gke

A secure, turn-key public Tezos baking service on Kubernetes
Apache License 2.0
32 stars 15 forks source link

Private node warning "p2p.maintenance: Too few connections (1)" [SOLVED] #37

Open denver-s opened 3 years ago

denver-s commented 3 years ago

For some reason, private-node failed to connect to public-node-0.

To fix this problem I connected to the gcloud shell:

gcloud container clusters get-credentials blockchain --region us-central1 --project <PROJECT_ID> && kubectl exec xtz-tezos-private-baking-node-mynode-<POD_ID> -c tezos-private-node --namespace tezos -it -- /bin/sh

Checked the p2p stat for the node:

tezos-admin-client -A xtz-tezos-private-baking-node-mynode p2p stat (-A is not used since version v8.0) tezos-admin-client -E http://xtz-tezos-private-baking-node-mynode:8732 p2p stat

GLOBAL STATS
  ↗ 43.58 MiB (24 B/s) ↘ 75.58 MiB (114 B/s)
CONNECTIONS
  ↗ idsxxxxxxxxxxxx 10.104.3.14:9732 (TEZOS_MAINNET.0 (p2p: 1)) 
KNOWN PEERS
  ⚏  1 idqyyyyyyyyyyy ↗ 0 B (0 B/s) ↘ 0 B (0 B/s)  
  ⚌  1 idsxxxxxxxxxxxx ↗ 358.83 kiB (24 B/s) ↘ 868.79 kiB (114 B/s)  
KNOWN POINTS
  ⚌  10.104.3.14:9732 idsxxxxxxxxxxxx 
  ⚏  x.y.z.k:9732
  ⚏  z.y.z.k:9732
  ⚏  j.y.z.k:9732
  ⚏  t.y.z.k:9732

So, the private node is connected to public-node-1 (10.104.3.14:9732) and the unconnected node is public-node-0 (10.104.2.14:9732).

tezos-admin-client -A xtz-tezos-private-baking-node-mynode connect address 10.104.2.14:9732 (-A is not used since version v8.0)

tezos-admin-client -E http://xtz-tezos-private-baking-node-mynode:8732 trust address 10.104.2.14:9732
tezos-admin-client -E http://xtz-tezos-private-baking-node-mynode:8732 connect address 10.104.2.14:9732

Now the p2p stat shows:

GLOBAL STATS
  ↗ 43.60 MiB (22 B/s) ↘ 75.62 MiB (28 B/s)
CONNECTIONS
  ↗ idqyyyyyyyyyyy 10.104.2.14:9732 (TEZOS_MAINNET.0 (p2p: 1)) 
  ↗ idsxxxxxxxxxxxx 10.104.3.14:9732 (TEZOS_MAINNET.0 (p2p: 1)) 
KNOWN PEERS
  ⚌  1 idqyyyyyyyyyyy ↗ 9.77 kiB (12 B/s) ↘ 10.90 kiB (17 B/s)  
  ⚌  1 idsxxxxxxxxxxxx ↗ 374.26 kiB (7 B/s) ↘ 906.24 kiB (9 B/s)

Hope it can help others!

nicolasochem commented 3 years ago

I have had this issue persistently; I normally bounce the private node to fix it. If you set up prometheus, you will normally get an alert when this happens.

So when I get an alert, I manually restart the node.

I even opened an issue on tezos gitlab related to this, feel free to update it with your details.

https://gitlab.com/tezos/tezos/-/issues/747