search

libp2p / js-libp2p-webrtc-direct

Dial using WebRTC without the need to set up any Signalling Rendezvous Point!
https://github.com/libp2p/js-libp2p-webrtc-direct
Other
78 stars 19 forks source link

Question. How to use with js-ipfs? #94

Closed vogdb closed 3 years ago

vogdb commented 3 years ago

Can you please advice on how to use it with an IPFS node in node.js? I tried this snippet:

const Libp2p = require('libp2p')
const IPFS = require('ipfs')
const MulticastDNS = require('libp2p-mdns')
const WebRTCDirect = require('libp2p-webrtc-direct')
const {NOISE} = require('libp2p-noise')

const libp2pBundle = (opts) => {
  const peerId = opts.peerId

  return new Libp2p({
    peerId,
    addresses: {
      listen: ['/ip4/127.0.0.1/tcp/0']
    },
    modules: {
      transport: [
        WebRTCDirect
      ],
      connEncryption: [
        NOISE
      ],
      peerDiscovery: [
        MulticastDNS,
      ],
    },
    config: {
      peerDiscovery: {
        bootstrap: {
          enabled: false,
        }
      }
    }
  })
}

async function main () {
  const node = await IPFS.create({
    repo: 'ipfs-' + Math.random(),
    libp2p: libp2pBundle
  })
}

document.addEventListener('DOMContentLoaded', async () => {
  main()
})

But received:

Uncaught (in promise) Error: no valid addresses were provided for transports [WebRTCDirect,Circuit]

The main IPFS node was started via jsipfs daemon:

vogdb@vogdb-IdeaPad-L340-17IRH-Gaming:~/workspace/xxx/ipfs-custom-lib2p$ jsipfs daemon
Initializing IPFS daemon...
js-ipfs version: 0.49.0
System version: x64/linux
Node.js version: 12.18.3
Swarm listening on /ip4/127.0.0.1/tcp/4002/p2p/QmR6vM94JXLwWqEwB6cCt16gpdunDtcassPyfNF2gC9c8o
Swarm listening on /ip4/192.168.1.42/tcp/4002/p2p/QmR6vM94JXLwWqEwB6cCt16gpdunDtcassPyfNF2gC9c8o
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/p2p/QmR6vM94JXLwWqEwB6cCt16gpdunDtcassPyfNF2gC9c8o
API listening on /ip4/127.0.0.1/tcp/5002/http
Gateway (read only) listening on /ip4/127.0.0.1/tcp/9090/http
Web UI available at http://127.0.0.1:5002/webui
Daemon is ready
vasco-santos commented 3 years ago

Hello @vogdb

The issue is that the code snippet his attempting to listen for tcp connections by:

addresses: {
   listen: ['/ip4/127.0.0.1/tcp/0']
},

You need to listen for webrtcDirect connections. You need to specify a webrtc-direct valid address like:

addresses: {
  listen: ['/ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct']
},

Let me know if you find any other issues. You need to also add the webrtcDirect transport to the js-ipfs daemon if you are looking to connect both through it

Out of this context, from the code snippet, it seems that you are tying to use libp2p-mdns within a browser context. This will not work as the mdns protocol is not available in a browser environment

vogdb commented 3 years ago

Thank you so much for the response. If there is documentation/example about integration of WebrtcDirect and js-ipfs, just please point me to it. Otherwise, I still have some questions:

You need to also add the webrtcDirect transport to the js-ipfs daemon if you are looking to connect both through it

How to do it? I don't see any mentioning of this in documentation of config paragraph https://github.com/ipfs/js-ipfs/blob/master/docs/CONFIG.md. Also the search does not give anything https://github.com/ipfs/js-ipfs/search?q=webrtcdirect.

Out of this context, from the code snippet, it seems that you are tying to use libp2p-mdns within a browser context. This will not work as the mdns protocol is not available in a browser environment

I hoped to establish Browser to Node.js connections among IPFS nodes. What would be your recommendation on that? Again if there is documentation somewhere, just point me. I couldn't find any.

Thank you again.

vasco-santos commented 3 years ago

There is no direct documentation on how to do this at the moment. If you are willing to help with that once you create your setup it would be super helpful.

The IPFS daemon itself is not easy to configure at this level currently. The ipfs-daemon package allows specifying the libp2p config https://github.com/ipfs/js-ipfs/blob/master/packages/ipfs-daemon/src/index.js#L46 but it seems that you would need to orchestrate the daemon with the cli+http server yourself. Anyway, I am not sure if supporting this type of interactions at the daemon level is something the project wants to support. I am not really into the ipfs setup in these scenarios.

So, I would like to understand your requirements and if you really want to use the daemon. The ipfs daemon would be useful if you expect to leverage its HTTP API + CLI while it is deployed somewhere. If you are not looking into this, I highly recommend you to use IPFS as a library instead like the above browser code snippet but for Node. You can also create an issue in js-ipfs if you like, in order to understand if supporting this type of customisation in the daemon is desirable and better guidelines on how to do it.

Taking into account that you use IPFS as a library, you would create two projects (browser + node) using IPFS+libp2p as dependencies, which allow you to provide your own libp2p bundle with webrtcDirect.

I hoped to establish Browser to Node.js connections among IPFS nodes. What would be your recommendation on that? Again if there is documentation somewhere, just point me. I couldn't find any.

Do you have any requirement on achieving this with WebRTC for that connectivity? The usual scenarios we currently use are:

The main reason for this is that other IPFS implementations (including go-ipfs) do not support webrtc yet. This would mean that you would only be able to connect with js-ipfs Node.js nodes, which is not desirable as your node network topology would be highly affected.

For websockets, you just need to use a websocket address in your IPFS config https://github.com/ipfs/js-ipfs/blob/master/docs/CONFIG.md#swarm . The caveat here is that your IPFS node will need to be behind an SSL certificate with dns because browser security policies do not allow websocket connections with bare websockets. With this in mind, you need to use https://github.com/ipfs/js-ipfs/blob/master/docs/CONFIG.md#announce where you will specify the dns based addresses like: dns4/example.libp2p.io/tcp/443/wss/p2p/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs65. It is also important pointing out that since libp2p-websockets@0.15 we do not support attempting to dial non dns+wss addresses because they do not work in production as mentioned. However, for local testing/experimentations, it is possible to use local addresses without setting up SSL+DNS. If you want to experiment with that look at https://github.com/libp2p/js-libp2p/blob/master/doc/migrations/v0.29-v0.30.md#development-and-testing

FYI: we will be working over the next months on improving the user experience of IPFS/libp2p in this side of things, aiming to have IPFS capable of having its own certificate running out of the box. But, this is not there yet.

Happy to discuss this further, either way you want to go with your work

vogdb commented 3 years ago

There is no direct documentation on how to do this at the moment. If you are willing to help with that once you create your setup it would be super helpful.

I will be happy to help. I am very fond of the project in general.

So, I would like to understand your requirements...

No need for the daemon. It's a hobby project aside of my main work (scientific Python stack). I want to create an app to transfer files between a mobile and a PC when they are on the same local network/WiFi router (possibly extend it later for Internet). For mobile I want to wrap a browser IPFS into Cordova/Capacitor. For PC I want to have an electron app with Node.js IPFS. I went for WebrtcDirect as it looked as a solution with minimum to manage. No need for Webrtc signaling server or SSL+DNS in Websocket's case.

I lack knowledge in telecommunications/networking. If you can advice on a better approach, I would really appreciate it, and of course I will be happy to update documentation of the project if necessary. The idea of IPFS is thrilling, the current corporate Internet disappoints me.

vasco-santos commented 3 years ago

No need for the daemon For mobile I want to wrap a browser IPFS into Cordova/Capacitor. For PC I want to have an electron app with Node.js IPFS. I went for WebrtcDirect as it looked as a solution with minimum to manage.

With the points mentioned I would recommend using IPFS as a library for Node.js as well: You would have something like:

node.js

const Libp2p = require('libp2p')
const IPFS = require('ipfs')
const WebRTCDirect = require('libp2p-webrtc-direct')
const Mplex = require('libp2p-mplex')
const {NOISE} = require('libp2p-noise')

const createLibp2p = (opts) => {
  const peerId = opts.peerId

  return new Libp2p({
    peerId,
    addresses: {
      listen: ['/ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct']
    },
    modules: {
      transport: [WebRTCDirect],
      streamMuxer: [Mplex],
      connEncryption: [NOISE]
    }
  })
}

// TODO: create IPFS with this function

browser.js

const Libp2p = require('libp2p')
const IPFS = require('ipfs')
const WebRTCDirect = require('libp2p-webrtc-direct')
const Mplex = require('libp2p-mplex')
const {NOISE} = require('libp2p-noise')
const Bootstrap = require('libp2p-bootstrap')

const createLibp2p = (opts, bootstrappers) => {
  const peerId = opts.peerId

  return new Libp2p({
    modules: {
      transport: [WebRTCDirect],
      streamMuxer: [Mplex],
      connEncryption: [NOISE],
      peerDiscovery: [Bootstrap]
    },
    config: {
      peerDiscovery: {
        [Bootstrap.tag]: {
          enabled: true,
          list: bootstrappers
        }
      }
    }
  })
}

// TODO: create IPFS with this function
// Note the bootstrappers parameter. It should be an array with the multiaddr of your node: ['/ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct']

With this, if you use IPFS's swarm API you will be able to log the connected peers and after both peers are running, they should be connected and they are ready to exchange files.

Let me know if you could connect both nodes. For moving this into the public network, you would need to have a public address for your server that could be reached from anywhere.

vogdb commented 3 years ago

Thank you so much that you find time to help me during New Eve's holidays!

...array with the multiaddr of your node: ['/ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct']

Can you please clarify what do you mean by that? Here is the code for node.js

const Libp2p = require('libp2p')
const IPFS = require('ipfs')
const WebRTCDirect = require('libp2p-webrtc-direct')
const Mplex = require('libp2p-mplex')
const {NOISE} = require('libp2p-noise')

const libp2pBundle = (opts) => {
  const peerId = opts.peerId

  return new Libp2p({
    peerId,
    addresses: {
      listen: ['/ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct']
    },
    modules: {
      transport: [WebRTCDirect],
      streamMuxer: [Mplex],
      connEncryption: [NOISE]
    }
  })
}

async function main () {
  const node = await IPFS.create({
    repo: 'ipfs-' + Math.random(),
    libp2p: libp2pBundle
  })
  console.log(`ipfs node's config: ${JSON.stringify(await node.config.get('Addresses'), null, 2)}`)

  setInterval(async () => {
    try {
      const peers = await node.swarm.peers()
      if (peers.length > 0)
        console.log(`The node now has ${peers.length} peers.`)
    } catch (err) {
      console.log('An error occurred trying to check out peers:', err)
    }
  }, 2000)
}

main()

It starts OK with the output:

node public/node-ipfs.js 
generating 2048-bit (rsa only) rsa keypair...
to get started, enter:

        jsipfs cat /ipfs/QmfGBRT6BbWJd7yUc2uYdaUZJBbnEFvTqehPFoSMQ6wgdr/readme

Swarm listening on /ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct/p2p/Qma5nB1MDePZoFcgztXGDYr9K7VdiVZhQrbd5cz9DEXvis
ipfs node's config: {
  "Swarm": [
    "/ip4/0.0.0.0/tcp/4002",
    "/ip4/127.0.0.1/tcp/4003/ws"
  ],
  "API": "/ip4/127.0.0.1/tcp/5002",
  "Gateway": "/ip4/127.0.0.1/tcp/9090",
  "Delegates": [
    "/dns4/node0.delegate.ipfs.io/tcp/443/https",
    "/dns4/node1.delegate.ipfs.io/tcp/443/https",
    "/dns4/node2.delegate.ipfs.io/tcp/443/https",
    "/dns4/node3.delegate.ipfs.io/tcp/443/https"
  ]
}

browser.js that I used:

const Libp2p = require('libp2p')
const IPFS = require('ipfs')
const WebRTCDirect = require('libp2p-webrtc-direct')
const Mplex = require('libp2p-mplex')
const {NOISE} = require('libp2p-noise')
const Bootstrap = require('libp2p-bootstrap')

const createLibp2p = (opts, bootstrappers) => {
  const peerId = opts.peerId

  return new Libp2p({
    peerId,
    modules: {
      transport: [WebRTCDirect],
      streamMuxer: [Mplex],
      connEncryption: [NOISE],
      peerDiscovery: [Bootstrap]
    },
    config: {
      peerDiscovery: {
        [Bootstrap.tag]: {
          enabled: false,
          list: ['/ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct/p2p/Qma5nB1MDePZoFcgztXGDYr9K7VdiVZhQrbd5cz9DEXvis']
        }
      }
    }
  })
}

async function main () {
  const node = await IPFS.create({
    repo: 'ipfs-' + Math.random(),
    lib2p: createLibp2p,
    // How to correctly set bootstrappers?
    // libp2p: (...args) => createLibp2p(...args, ['/ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct/p2p/Qma5nB1MDePZoFcgztXGDYr9K7VdiVZhQrbd5cz9DEXvis']),
    // Bootstrap: ['/ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct/p2p/Qma5nB1MDePZoFcgztXGDYr9K7VdiVZhQrbd5cz9DEXvis']
  })

  setInterval(async () => {
    try {
      const peers = await node.swarm.peers()
      console.log(`The node now has ${peers.length} peers.`)
      console.log(`peers: ${JSON.stringify(peers, null, 2)}`)
    } catch (err) {
      console.log('An error occurred trying to check our peers:', err)
    }
  }, 30000)
}

document.addEventListener('DOMContentLoaded', async () => {
  main()
})
vasco-santos commented 3 years ago

Sorry. I missed the notification for the latest message.

server.js

const Libp2p = require('libp2p')
const IPFS = require('ipfs')
const Bootstrap = require('libp2p-bootstrap')
const WebRTCDirect = require('libp2p-webrtc-direct')
const Mplex = require('libp2p-mplex')
const {NOISE} = require('libp2p-noise')

const libp2pBundle = (opts) => {
  const peerId = opts.peerId

  return new Libp2p({
    peerId,
    addresses: {
      listen: ['/ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct']
    },
    modules: {
      transport: [WebRTCDirect],
      streamMuxer: [Mplex],
      connEncryption: [NOISE]
    },
   config: {
      peerDiscovery: {
        [Bootstrap.tag]: {
          enabled: false,
        }
      }
    }
  })
}

async function main () {
  const node = await IPFS.create({
    repo: 'ipfs-' + Math.random(),
    libp2p: libp2pBundle
  })
  console.log(`ipfs node's id: ${JSON.stringify(await node.id()}`)

  setInterval(async () => {
    try {
      const peers = await node.swarm.peers()
      if (peers.length > 0)
        console.log(`The node now has ${peers.length} peers.`)
    } catch (err) {
      console.log('An error occurred trying to check out peers:', err)
    }
  }, 2000)
}

main()

Just changed bootstrap discovery to false and the node log in the context of this experiment. If you want your node to connect to the public network, you should not disable the bootstrap, but in this case it will be easy to see what other nodes connect, as the ones connecting with you were bootstrap nodes.

Moving into your questions, the bootstrap address is good! /ip4/127.0.0.1/tcp/9090/http/p2p-webrtc-direct/p2p/Qma5nB1MDePZoFcgztXGDYr9K7VdiVZhQrbd5cz9DEXvis It should always be the same as presented in Node.js node logs as Swarm listening on .... You don't need to move them from the function, and can have them as you did.

The problem to not get the connection might be the bootstrap set to false. Sorry, I did a copy paste and forgot to change it to true. And also you did good in adding the peerId.

Let me know if you got this to work with bootstrap enabled. If not, do you have this demo in any repo I can try?

vogdb commented 3 years ago

I tried your suggestions, I still can't see the connection. The repo is here https://github.com/vogdb/ipfs-custom-lib2p. Readme contains the recipe, I use to test. Please let me know if anything to add there.

vasco-santos commented 3 years ago

Thanks for reaching out @vogdb

I looked into this, and while there is an issue with your code, I also found some bugs on our code that I created PRs to fix (you can see them linked above this comment). I will create a PR to fix your module for now

EDIT: https://github.com/vogdb/ipfs-custom-lib2p/pull/1

vogdb commented 3 years ago

Thank you for your changes. A couple of remaining questions from me.

There is no direct documentation on how to do this at the moment. If you are willing to help with that once you create your setup it would be super helpful.

vasco-santos commented 3 years ago

How you recommend to update documentation? Should I create a separate page in this repo? In js-ipfs repo? Or write more documentation in ipfs-custom-lib2p and move it under https://github.com/ipfs as an example?

Probably an example in the libp2p repo would be the ideal https://github.com/libp2p/js-libp2p/tree/master/examples . While you are using IPFS here, the main components used are libp2p. You can also extend the IPFS example https://github.com/ipfs/js-ipfs/tree/master/examples/custom-libp2p to reference this example, or missing bits to how to configure libp2p within IPFS.

Is there a way to avoid this long hash QmPj9ZZ6notLfV9khV1FtxH1Goe5sVaUyqgoXrTYQWp382? Or at least shorten it to 6 symbols max, so a user can type it.

The hash is a cryptographically generated identifier from the Peer key pair. This way, it is not possible to shorten it. I understand your point, but this is something needed

vogdb commented 3 years ago

Thank you for clarifying. I'm waiting then for #97. After its merge and release, I will update the documentation.

vasco-santos commented 3 years ago

@vogdb closing this issue as I am releasing a new version of the module with the fix now. Feel free to re-open if further issues appear

vogdb commented 3 years ago

I've just checked. It works. Thank you. One thing to notice is that it didn't work in Firefox 84.0.2 (64-bit) Ubuntu Linux. Worked in Chromium Version 88.0.4324.96 (Official Build) snap (64-bit).

vasco-santos commented 3 years ago

One thing to notice is that it didn't work in Firefox 84.0.2 (64-bit) Ubuntu Linux. Worked in Chromium Version 88.0.4324.96 (Official Build) snap (64-bit).

Thanks for this feedback. Can you open a new issue with this information so that I can have a look soon

vogdb commented 3 years ago

@vasco-santos sorry for pinging here. I'm trying to find a company that works with IPFS/Libp2p. Can you recommend one? So far, I could find only Textile. I'm thinking to switch to IPFS/Libp2p fulltime.

vasco-santos commented 3 years ago

Hey @vogdb You can reach out to andre@moxy.studio via email. I already talked with him

vogdb commented 3 years ago

Thank you!