search

mxinden / kademlia-exporter

Libp2p Kademlia Exporter exposing Prometheus metrics
MIT License
21 stars 6 forks source link

Monitor subspace network #274

Open mxinden opened 1 year ago

mxinden commented 1 year ago

@nazar-pc I have configured kademlia-exporter.max-inden.de/ to monitor the subspace network via the bootstrap nodes you provided.

You pointed me to this Kademlia protocol name, namely /subspace/kad/0.1.0.

https://github.com/subspace/subspace/blob/main/crates/subspace-networking/src/create.rs#L44

I don't think this is accurate:

libp2p-lookup direct --address /dns/bootstrap-0.gemini-3c.subspace.network/tcp/30333/p2p/12D3KooWF9LjJX8XSwh3WfyyZYT7TepkW84fV6LLidUBprXjmgZ9
Local peer id: 12D3KooWJJzDGHYAN3SG87FqDjn9WiQvsaCXzGLAVC6iejogLZU7
Lookup for peer with id PeerId("12D3KooWF9LjJX8XSwh3WfyyZYT7TepkW84fV6LLidUBprXjmgZ9") succeeded.

Protocol version: "/substrate/1.0"
Agent version: "Subspace/v0.1.0-a941651f6a2e6685b7db16c3286a440c6bddbf2a (light-breath-4078)"
Listen addresses:
        - "/dns/bootstrap-0.gemini-3c.subspace.network/tcp/30333"
        - "/ip4/65.108.228.83/tcp/30333"
        - "/ip4/127.0.0.1/tcp/30333"
        - "/ip4/172.25.0.2/tcp/30333"
Protocols:
        - "/ab946a15b37f59c5f4f27c5de93acde9fe67a28e0b724a43a30e4fe0e87246b7/block-announces/1"
        - "/subspace-gemini-3c/block-announces/1"
        - "/ab946a15b37f59c5f4f27c5de93acde9fe67a28e0b724a43a30e4fe0e87246b7/transactions/1"
        - "/subspace-gemini-3c/transactions/1"
        - "/subspace/cross-domain-messages"
        - "/ipfs/ping/1.0.0"
        - "/ipfs/id/1.0.0"
        - "/ipfs/id/push/1.0.0"
        - "/ab946a15b37f59c5f4f27c5de93acde9fe67a28e0b724a43a30e4fe0e87246b7/kad"
        - "/subspace-gemini-3c/kad"
        - "/ab946a15b37f59c5f4f27c5de93acde9fe67a28e0b724a43a30e4fe0e87246b7/light/2"
        - "/subspace-gemini-3c/light/2"
        - "/ab946a15b37f59c5f4f27c5de93acde9fe67a28e0b724a43a30e4fe0e87246b7/state/2"
        - "/subspace-gemini-3c/state/2"
        - "/ab946a15b37f59c5f4f27c5de93acde9fe67a28e0b724a43a30e4fe0e87246b7/sync/2"
        - "/subspace-gemini-3c/sync/2"

I think it should be /subspace-gemini-3c/kad. Can you confirm?

nazar-pc commented 1 year ago

Depends on which network we're talking about. Currently we have dual stack: blockchain nodes (Substrate-based with corresponding networking inherited) and Subspace DSN (separate libp2p networking stack we're developing that'll replace Substate's at some point).

For Substrate it is /subspace-gemini-3c/kad, for Subspace (same identity, runs on port 30433) it is /subspace/kad/0.1.0. We have more nodes on Subspace DSN due to both nodes and farmers joining it (and various clients in the future to retrieve data), while Substate is just for blockchain nodes.

mxinden commented 1 year ago

@nazar-pc I am unable to connect to 30433:

➜  subspace git:(master) ✗ RUST_LOG=debug libp2p-lookup direct --address /dns/bootstrap-2.gemini-3c.subspace.network/tcp/30433/p2p/12D3KooWM4NyQbGQ395o7UTsd7pZ64bne1kin3Hhe5VZvFXa7SzX
Local peer id: 12D3KooWCRZCK2Lr4rQtgi6r9XYEPyR9ztDcZoVkTtFHPoFcgfoq
[2023-03-23T10:30:57Z DEBUG libp2p_dns] Dialing /ip4/65.108.232.57/tcp/30433/p2p/12D3KooWM4NyQbGQ395o7UTsd7pZ64bne1kin3Hhe5VZvFXa7SzX
[2023-03-23T10:30:57Z DEBUG libp2p_tcp] dialing 65.108.232.57:30433
[2023-03-23T10:31:17Z ERROR libp2p_lookup] Lookup failed: Timeout.

Port seems to be open though:

➜  subspace git:(master) ✗ sudo nmap -p- 65.108.228.83
Starting Nmap 7.92 ( https://nmap.org ) at 2023-03-23 11:28 CET
Nmap scan report for static.83.228.108.65.clients.your-server.de (65.108.228.83)
Host is up (0.052s latency).
Not shown: 65532 closed tcp ports (reset)
PORT      STATE    SERVICE
22/tcp    open     ssh
30333/tcp open     unknown
30433/tcp filtered unknown

Nmap done: 1 IP address (1 host up) scanned in 97.44 seconds

Any pointers?

nazar-pc commented 1 year ago

Well, we have seen it a few times: libp2p port just stops responding after some time. We'll fix it soon with software upgrade and restart.

nazar-pc commented 1 year ago

We have upgraded our network from Gemini 3c to Gemini 3d.

Substrate network endpoints:

/dns/bootstrap-0.gemini-3d.subspace.network/tcp/30333/p2p/12D3KooWG4rXVY4Z6rv2ZTqw1debMQVeGNUEFvDNkEXLTmKbqAg6
/dns/bootstrap-1.gemini-3d.subspace.network/tcp/30333/p2p/12D3KooWN4Uu1Lb8p6skmMNFYYKksQdjQXh7Nvbq5jxXGr5gDQyF

Subspace network endpoints:

/dns/bootstrap-0.gemini-3d.subspace.network/tcp/30433/p2p/12D3KooWG4rXVY4Z6rv2ZTqw1debMQVeGNUEFvDNkEXLTmKbqAg6
/dns/bootstrap-1.gemini-3d.subspace.network/tcp/30433/p2p/12D3KooWN4Uu1Lb8p6skmMNFYYKksQdjQXh7Nvbq5jxXGr5gDQyF
/dns/bootstrap-0.gemini-3d.subspace.network/tcp/50000/p2p/12D3KooWRNoaozU9DuRRWq7cMPYtcmGvXSjsxKwvTyv49FVhctQA
/dns/bootstrap-1.gemini-3d.subspace.network/tcp/50000/p2p/12D3KooWBLiyiL1iwTCUx5jzbYZy1JziyPJsV6hvpe4YbufaQp65

Those with port 30433 can be down, we suspect Kademlia's provider handling takes too much time and keeps the thread occupied and unable to handle incoming requests fast enough (we are going to replace that with custom request/response protocol soon), but those on port 50000 should be up since they are purpose-built to be lightweight bootstrap nodes and don't handle any data storage or anything like that.

nazar-pc commented 9 months ago

We have launched incentivized Gemini 3f testnet last week, here are up to date endpoints for it.

Substrate:

/dns/bootstrap-0.gemini-3f.subspace.network/tcp/30333/p2p/12D3KooWEcKMS2XtLfPxuMYkMzhZwRMuQ9m37t41MgA4cyTK98G7
/dns/bootstrap-1.gemini-3f.subspace.network/tcp/30333/p2p/12D3KooWJtnuBSL5WqwaxNbgssbkznz97vVmMRZ87cVD22tTW6Z4

Subspace:

/dns/bootstrap-0.gemini-3f.subspace.network/tcp/30533/p2p/12D3KooWCahCdQpYGnEKzubYJbJAp4U2F8bMERdtwx6SWQPxoRRv
/dns/bootstrap-1.gemini-3f.subspace.network/tcp/30533/p2p/12D3KooWPkFWWhMXEzKu1k3d6iRea56LkEJvf2EstD7Byjp9vxKZ

For Subspace protocol version on Identify should be /subspace/92e91e657747c41eeabed5129ff51689d2e935b9f6abfbd5dfcb2e1d0d035095.