FindProviders cannot find providers of given CID

HoseinHaqiqian commented 1 year ago

Version: ^0.45.9
Platform: Darwin arm64

Severity: High

Description:

What you did: i am tried to find advertised cid via findProviders method.
What happened : I can successfully advertise the CID, but cannot retrieve it with FindProviders function in sender criteria
What you expected to happen: return Providers of given CID

Steps to reproduce the error:

DHT Config

const mDht = kadDHT({
    clientMode: true,
    maxInboundStreams: 100,
    maxOutboundStreams: 100,
    querySelfInterval: 10,
    // allowQueryWithZeroPeers: true,
  });

Libp2p configs

opts.connectionManager = {
    maxConnections: 250,
    minConnections: 100,
    dialTimeout: 10e3,
    maxParallelDials:10,
  };

opts.peerDiscovery = [
    bootstrap({
      list: StaticRelays(),
      timeout: 1000, // in ms,
      tagName: "bootstrap",
      tagValue: 50,
      tagTTL: 120000, // in ms
    }),
    mdns({interval : 1000 * 30}),
  ];

opts.services = {
    identifyService: identifyService(),
    dht: mDht,
  };

opts.streamMuxers = [yamux()];
  opts.transports = [tcp(), webSockets(), webTransport()];

  opts.connectionEncryption = [noise()];

try {
    host = await createLibp2p(opts);
    await host.start();
    await (host.services.dht as DualKadDHT).refreshRoutingTable();
  } catch (error) {
    throw Error(`Error with creating libp2p ${error}`);
  }

Static Relays

export function StaticRelays(): string[] {
  const staticRelays = [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/ipfs/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/ipfs/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
  ];

  return staticRelays;
}

The code i use to advertise cid await node.host.contentRouting.provide(ns);

The code i use to findProviders of cid

 const peersFound = await all(
      node.dht.findProviders(ns,{queryFuncTimeout:5000})
    );

HoseinHaqiqian commented 1 year ago

it is not working all the time. just sometimes work. it is not reliable.

HoseinHaqiqian commented 1 year ago

no updates?

maschad commented 1 year ago

Could you provide the output of the debug logs when you are unable to retrieve the CID?

HoseinHaqiqian commented 1 year ago

@maschad all logs ? DEBUG="libp2p:*" you mean?

maschad commented 1 year ago

@maschad all logs ? DEBUG="libp2p:*" you mean?

Yes no filter is necessary, but specifically the logs which occurred around the error thrown by findProviders

HoseinHaqiqian commented 1 year ago

The logs were more than 100,000 lines. There is no specific error, it just tries to find until i abort the mission. It keeps printing this line for finding bafkreibdfwenbvckwhxdpmm54ybnvj7bpznz7frs5kktrr3aon3fpsxjky

The cid mentioned above is what i am looking for.

libp2p:kad-dht:lan:content-routing Found 0 provider entries for bafkreibdfwenbvckwhxdpmm54ybnvj7bpznz7frs5kktrr3aon3fpsxjky and 0 closer peers

The lines around the above code are :


2023-07-24T21:24:26.269Z libp2p:identify identify completed for peer 12D3KooWE8Tcqh5rq2TB23AN3rfqKGitQSkqvLeQyvHEYvypZQxN and protocols [ '/ipfs/ping/1.0.0', '/libp2p/circuit/relay/0.2.0/stop', '/ipfs/lan/kad/1.0.0', '/libp2p/autonat/1.0.0', '/ipfs/id/1.0.0', '/ipfs/id/push/1.0.0', '/meshsub/1.1.0', '/meshsub/1.0.0', '/floodsub/1.0.0', '/libp2p/fetch/0.0.1', '/ipfs/bitswap/1.2.0', '/ipfs/bitswap/1.1.0', '/ipfs/bitswap/1.0.0', '/ipfs/bitswap', '/x/', '/libp2p/dcutr', '/libp2p/circuit/relay/0.2.0/hop', '/ipfs/kad/1.0.0' ]
2023-07-24T21:24:26.269Z libp2p:identify our observed address is /ip4/31.187.78.188/tcp/59478
2023-07-24T21:24:26.269Z libp2p:identify received identify from 12D3KooWE8Tcqh5rq2TB23AN3rfqKGitQSkqvLeQyvHEYvypZQxN
2023-07-24T21:24:26.269Z libp2p:identify received signedPeerRecord in push from 12D3KooWE8Tcqh5rq2TB23AN3rfqKGitQSkqvLeQyvHEYvypZQxN
2023-07-24T21:24:26.270Z libp2p:kad-dht:topology-listener:lan observed peer 12D3KooWE8Tcqh5rq2TB23AN3rfqKGitQSkqvLeQyvHEYvypZQxN with protocol /ipfs/lan/kad/1.0.0
2023-07-24T21:24:26.270Z libp2p:circuit-relay:transport:reservation-store add relay 12D3KooWE8Tcqh5rq2TB23AN3rfqKGitQSkqvLeQyvHEYvypZQxN
2023-07-24T21:24:26.270Z libp2p:circuit-relay:transport:reservation-store already have enough discovered relays
2023-07-24T21:24:26.270Z libp2p:kad-dht:lan peer 12D3KooWE8Tcqh5rq2TB23AN3rfqKGitQSkqvLeQyvHEYvypZQxN connected with protocols [
  '/floodsub/1.0.0',
  '/ipfs/bitswap',
  '/ipfs/bitswap/1.0.0',
  '/ipfs/bitswap/1.1.0',
  '/ipfs/bitswap/1.2.0',
  '/ipfs/id/1.0.0',
  '/ipfs/id/push/1.0.0',
  '/ipfs/kad/1.0.0',
  '/ipfs/lan/kad/1.0.0',
  '/ipfs/ping/1.0.0',
  '/libp2p/autonat/1.0.0',
  '/libp2p/circuit/relay/0.2.0/hop',
  '/libp2p/circuit/relay/0.2.0/stop',
  '/libp2p/dcutr',
  '/libp2p/fetch/0.0.1',
  '/meshsub/1.0.0',
  '/meshsub/1.1.0',
  '/x/'
]
2023-07-24T21:24:26.270Z libp2p:kad-dht:lan:routing-table added 12D3KooWE8Tcqh5rq2TB23AN3rfqKGitQSkqvLeQyvHEYvypZQxN with kad id AU7pafEesbetjCdHS67idp6Zi6anM6pXrCW2NWRd7bFw
2023-07-24T21:24:26.271Z libp2p:kad-dht:lan:query:12D3KooWQbZs8euoPVE2WACyo4Smmo6niEV8KttHyPangZL6YJqV queue idle
2023-07-24T21:24:26.271Z libp2p:kad-dht:lan:query:QmNLezABjmsxb5EHyTvX5PcoBkntsvgzFi6vTG7iQdXGL7 queue idle
2023-07-24T21:24:26.277Z libp2p:kad-dht:lan:query:QmQhyFSjaxfPMDPvarzRKpYGMK2yRo4FfDYKhbAUkyPjsB queue idle
2023-07-24T21:24:26.277Z libp2p:kad-dht:lan:content-routing Found 0 provider entries for bafkreibdfwenbvckwhxdpmm54ybnvj7bpznz7frs5kktrr3aon3fpsxjky and 0 closer peers
2023-07-24T21:24:26.277Z libp2p:kad-dht:wan:query-self run self-query, look for 20 peers timing out after 5000ms
2023-07-24T21:24:26.277Z libp2p:kad-dht:wan:peer-routing getClosestPeers to 12D3KooWQbZs8euoPVE2WACyo4Smmo6niEV8KttHyPangZL6YJqV
2023-07-24T21:24:26.277Z libp2p:kad-dht:wan:query:12D3KooWQbZs8euoPVE2WACyo4Smmo6niEV8KttHyPangZL6YJqV query:start
2023-07-24T21:24:26.278Z libp2p:kad-dht:wan:peer-routing closerPeersSingle aasaqaisednzp6t53nvtilgkcvn4gcar6hwfgnvdbrobh4mv73m4sb5p5xi6q from 12D3KooWPz2PfZwr9CBajAK42Vt5YW23oBc56Znctpd2gRKqHnwz
2023-07-24T21:24:26.278Z libp2p:kad-dht:wan:network sending FIND_NODE to 12D3KooWPz2PfZwr9CBajAK42Vt5YW23oBc56Znctpd2gRKqHnwz
2023-07-24T21:24:26.278Z libp2p:kad-dht:wan:peer-routing closerPeersSingle aasaqaisednzp6t53nvtilgkcvn4gcar6hwfgnvdbrobh4mv73m4sb5p5xi6q from 12D3KooWHCH11aWDi6n1Ctipzi7nr3ZesegXU7e3z7XAc4GUi9Uy
2023-07-24T21:24:26.278Z libp2p:kad-dht:wan:network sending FIND_NODE to 12D3KooWHCH11aWDi6n1Ctipzi7nr3ZesegXU7e3z7XAc4GUi9Uy

HoseinHaqiqian commented 1 year ago

am i doing something wrong? for example, should i announce local address before advertising? or something like that with addresses?

HoseinHaqiqian commented 1 year ago

i assume that main problem might be with adverstising as FindProviders is putting its best effort to find given cid.

HoseinHaqiqian commented 1 year ago

i even tried advertising same node with go written code (which is working fine) and find it with js, but it fails. the vice versa approach fails too.

maschad commented 1 year ago

i assume that main problem might be with adverstising as FindProviders is putting its best effort to find given cid.

This is my assumption as well, it's difficult to debug these types of issues without some sort of visualization as Kademlia determines the proximity of two keys based on the prefix length of the keys output by the hashing algorithm which at the genesis would have been dependent on the node's peerID.

Are any of the bootstrap nodes connected to the Peer that's advertising the CID?

HoseinHaqiqian commented 1 year ago

@maschad Yes. i am using default bootstraps which is working fine on go. /ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ I know that autoDial is on by default, but i still try to dial this node first. then try to advertise my cid. Now i am trying to implement my own relay and connect to it. might be problem solving. As it is okay when i advertise my cid on local, the issue most probably is because that nodes are behind NAT.

i can give you the example code i am using if you have time to investigate the problem.

maschad commented 1 year ago

Now i am trying to implement my own relay and connect to it. might be problem solving. As it is okay when i advertise my cid on local, the issue most probably is because that nodes are behind NAT.

It seems that the node trying to retrieve the CID wasn't connected to /ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ and that's resolved once you've overcome NAT Traversal. That's unrelated to this issue so I am going to close this for now. Feel free to open another issue if you are facing problems related to peer routing.

libp2p / js-libp2p