search

TechnitiumSoftware / Mesh

A secure, anonymous, peer-to-peer, instant messenger!
https://mesh.im
GNU General Public License v3.0
204 stars 57 forks source link

Does it uses STUN/TURN servers or any pointing middle server? #6

Open vini-brito opened 3 years ago

vini-brito commented 3 years ago

Hello! Many systems that uses DHT, like Jami, end up using signaling servers, does this software also uses them? If not, how do one peer discovers another? Asking because I'm also have a lot of interest in p2p apps. Big thanks! 😊

ShreyasZare commented 3 years ago

Hi. Thanks for asking. Mesh uses DHT where in each node needs to find out address of at least one node in the network to bootstrap. To do that, there is a simple API hosted on mesh.im that provides connectivity check (to let the app know its external end point and to test if inbound connections work) and since all nodes do connectivity check, the API also returns a list of last known nodes.

Once a node finds out other nodes, it stores that list and uses it again the next time the app starts in hope that one of these nodes should be online so that app can join back the DHT network.

If 2 nodes that are behind NAT/firewall want to join a chat then they use TCP relays. TCP relays are the DHT nodes which can accept incoming connections. So, any node that has public IP address becomes a TCP relay. A node selects 3 random relays and advertises their chat network ID using these relays using which then can accept inbound connections.

Over LAN networks, a node uses broadcast beacon when it starts to let other nodes know its presence. The other nodes then start introducing themselves to the new node to form a DHT in that network segment.

The protocol uses TCP for all interaction and a single TCP channel is used for all purposes by multiplexing it similar to how HTTP/2 works.

vini-brito commented 3 years ago

Awesome, thanks for the thorough explanation! Success! Oh, if I may, are there future plans for a linux client?

ShreyasZare commented 3 years ago

Awesome, thanks for the thorough explanation! Success! Oh, if I may, are there future plans for a linux client?

Yes, there is a plan to convert the core app into a console app with a built in local web server which will be cross platform. The UI can then be accessed using any available web browser.

shubhamdhanetia commented 3 years ago

Hello @ShreyasZare , I am new to P2P protocol. I have some questions if you can guide. I wanted to know if two different Private networks can connect over chat or they must need to be under same Private Network like Router? I have tested this application. Works well when the nodes are connected in same router. But when I tried to use mobile internet hotspot, it couldn't work.

ShreyasZare commented 3 years ago

I wanted to know if two different Private networks can connect over chat or they must need to be under same Private Network like Router?

If two nodes are on different private networks and if there is another 3rd node which happens to be on both networks then the 2 nodes will be able to connect via the 3rd node.

I have tested this application. Works well when the nodes are connected in same router. But when I tried to use mobile internet hotspot, it couldn't work.

Yes, on LAN networks its quite straight forward and it always work reliably. On mobile Internet networks, I have observed that a lot of these mobile ISPs block incoming connection requests for reasons unknown to me. This affects p2p apps greatly and requires the DHT network to have at least one node available which can accept incoming connections to allow other nodes behind firewall to connect with each other.

shubhamdhanetia commented 3 years ago

Thanks @ShreyasZare for the quick response. "If two nodes are on different private networks and if there is another 3rd node which happens to be on both networks then the 2 nodes will be able to connect via the 3rd node."

ShreyasZare commented 3 years ago

Thanks @ShreyasZare for the quick response. "If two nodes are on different private networks and if there is another 3rd node which happens to be on both networks then the 2 nodes will be able to connect via the 3rd node."

* I am confused on this part. For example, I want to connect to a person over the internet (Different Private Network). What steps needs to be followed ? So there are two nodes in picture, me and that person. How can I have 3rd node which would be connected to both me and that person. Is there something which we needs to do out of the box which is not supported directly by this application ?

The design of Mesh makes an assumption that there will be at least one node with the ability to accept inbound connections. So, the 3rd node will be some unknown random user.

Since Mesh is not very popular, most of the times there is no such node online to help other nodes behind firewall or NAT. And most people who have home broadband tend to either not have public IP address, are behind a firewall, or don't know how to enable UPnP on their routers to allow Mesh to do automatic port forwarding for receiving incoming connections.

ShreyasZare commented 3 years ago

Adding to last comment: IPv6 is expected to make it much better for p2p apps to work but right not most of the ISPs providing IPv6 are mobile operators and they have firewall blocking inbound connections.

shubhamdhanetia commented 3 years ago

Oh Okay. So is there any Hack we can do to test it out using VPN or something. I have one extra system with me available. Can I use it as 3rd node between me and the person over the internet using proxy server like VPN or any other thing ?

And Just a thought, if I purchase a windows instance over the cloud, then install this application there. Then I will create Anonymity profile on the server. Then we both needs to connects to that server using the User Id of the server profile. Once done, we internally would able to connect right ?

And using just UserID of the server user profile, would I be able to connect easily ? or I need to put IP address of the server somewhere ?

And also would there be any firewall issue arise if we try to connect to windows server using this Application?

ShreyasZare commented 3 years ago

Oh Okay. So is there any Hack we can do to test it out using VPN or something. I have one extra system with me available. Can I use it as 3rd node between me and the person over the internet using proxy server like VPN or any other thing ?

If you setup your own VPN server which supports bridging (to allow broadcast traffic over VPN) and both of you connect to the same VPN then it would work similar to how it works over LAN.

And Just a thought, if I purchase a windows instance over the cloud, then install this application there. Then I will create Anonymity profile on the server. Then we both needs to connects to that server using the User Id of the server profile. Once done, we internally would able to connect right ?

You can setup Mesh to run on cloud server with a p2p profile and it will work automatically. As the cloud Mesh instance does connectivity check with mesh.im, its IP will get registered for few mins. So, when you start Mesh on your computer, it will automatically find the cloud server and add it as a DHT node.

Only concern will be that your cloud server running the Mesh node is public and so not just you but anyone using Mesh will be able to use it automatically. So, your cloud server will route data for other people who are behind firewall/NAT too.

And using just UserID of the server user profile, would I be able to connect easily ? or I need to put IP address of the server somewhere ?

Its done automatic using DHT network. A node just needs to know one node that is part of the DHT network to join the same network. Nodes will find out each other via DHT using the connectivity check service running on mesh.im for initial introduction.

And also would there be any firewall issue arise if we try to connect to windows server using this Application?

On cloud servers there is no issue with firewalls since they are used to host server applications and Mesh automatically adds a firewall rule to allow itself.

Check the Network Info window in Mesh on the cloud server to see if IPv4 External End Point is displayed. The external end point is where the Mesh app will accept inbound connections.

shubhamdhanetia commented 3 years ago

Thanks, @ShreyasZare for clearing my doubts. So, we can use P2P profiles for both LAN and over the internet. 1) If I have a basic cloud server, around how many peoples it can help throughout the internet to establish a connection between them. 2) Mesh.im works similarly to BitTorrent tracker which also helps peers to find each other? 3) If I establish a connection with a peer and there is no cloud server left after it, so would I still able to remain connected with the peer. 4) As you said, when we run mesh on a cloud instance, its IP is added to the service running on mesh.im. So, if mesh.im service is down, even though there are 100s of the cloud server available, a new mesh node won't be able to find the cloud servers? So is it like single point of failure ? Would we need multiple instances of the mesh.im service for reliability? 5) Is the service just run on the mesh.im or it runs similarly on each mesh node? Just to make sure there is no central dependency. 6) Do the mesh node on our computer keep tracking for available cloud servers and updating the DHT node from time to time ?

ShreyasZare commented 3 years ago

So, we can use P2P profiles for both LAN and over the internet.

Yes, p2p profiles will work on any network. It can also work on LAN networks with no Internet access.

  1. If I have a basic cloud server, around how many peoples it can help throughout the internet to establish a connection between them.

Cloud servers have good bandwidth so a single server can serve 10s or even 100s of mesh nodes for text chat that are tunneled through it. File sharing feature will consume bandwidth quickly though so not many concurrent file transfers would work via single server.

But the good thing is that not everyone depends on the could server if they have public IP address on their router with UPnP enabled.

  1. Mesh.im works similarly to BitTorrent tracker which also helps peers to find each other?

Yes, its kind of like a BitTorrent tracker but its actually a connectivity check service. Each node uses it to see if they are able to accept inbound connections from the Internet. If the connectivity service is able to test successfully that a node is reachable, it keeps that node's external end points in memory for some time to share it with any other node that tries to do connectivity check.

  1. If I establish a connection with a peer and there is no cloud server left after it, so would I still able to remain connected with the peer.

If both the peers are behind NAT/Firewall then they are essentially tunneling data via the cloud server. So if the server goes down then the connection between both will be lost.

If there are more than one nodes that can accept inbound connections then both the peers will find another node to connect automatically.

In the Network Info window, you can see the TCP relays listed. Mesh node tries to find at least 3 such TCP relays and keeps a connection with them. These TCP relays are nothing but the other random nodes that are able to accept inbound connections.

  1. As you said, when we run mesh on a cloud instance, its IP is added to the service running on mesh.im. So, if mesh.im service is down, even though there are 100s of the cloud server available, a new mesh node won't be able to find the cloud servers? So is it like single point of failure ? Would we need multiple instances of the mesh.im service for reliability?

Yes, that is correct. New nodes will not be able to join the DHT network without accessing mesh.im. This is the bootstrap problem with any DHT network as you need to know at least one node to find out the other nodes in the network.

Nodes also save the list of other nodes in DHT just in case if they come back online and mesh.im is down for any reason hoping that at least one of the saved node will still be online and will allow joining the DHT network back.

There is a feature planned for this scenario such that you can manually add a node's external end point to your Mesh's DHT. So even if mesh.im is down, you can find out at least one node end point via other means and join the DHT network.

  1. Is the service just run on the mesh.im or it runs similarly on each mesh node? Just to make sure there is no central dependency.

There is no node maintained on mesh.im, it only provides connectivity check service that helps introduce to nodes that also did connectivity check. Anything other than that is purely p2p.

  1. Do the mesh node on our computer keep tracking for available cloud servers and updating the DHT node from time to time ?

Mesh node does not know if the other node is cloud server or just another user. It just keep refreshing DHT network and if its able to connect to a node on the DHT network then it keeps a note of it and then randomly select 3 such nodes to act as TCP relays. All nodes keep track of at least 3 TCP relays even if they are able to accept incoming connections for redundancy.

Thanks, @ShreyasZare for clearing my doubts.

You're welcome.

shubhamdhanetia commented 3 years ago

"File sharing feature will consume bandwidth quickly though so not many concurrent file transfers would work via the single server"

ShreyasZare commented 3 years ago

It seems your are super confused!

"File sharing feature will consume bandwidth quickly though so not many concurrent file transfers would work via the single server"

That was regarding if you decide to run a cloud server with Mesh node running so that you and your peers can connect through being behind NAT/Firewall. This solution is only required since Mesh is not being used by many people and there are no nodes online for long hours.

If you have public IP address then running this cloud server is totally not necessary. If you get a public IP address from your broadband provider then you can talk using Mesh with literally anyone on the Internet. Just ask your broadband provider for public IP and they will assign it for something like INR 100/month which is very cheap compared to buying a cloud server.

  • But BitTorrent protocol designed to get rid of bandwidth issues at the server-side. The BitTorrent tracker helps to get two peers connected so that they can share the data internally without needing the middle tunnel right to save bandwidth? If I am using a BitTorrent application to download a file and I am using private internet having NAT, so am I downloading the files via a tunnel or directly through a peer, and at the same time, a peer is getting file directly through me or via a tunnel?

For BitTorrent to work, the torrent must have at least one seeder. One can become seeder only if they can accept inbound connections. Without a seeder, you wont be able to download the torrent.

This is same with the case of Mesh. Since you and your peer are on private networks and cant run on a public IP so hosting a node on cloud server will add a node to the Mesh DHT network which then will be available to everyone to use when they are behind NAT/Firewall.

  • Can't we have completely peer-to-peer decentralization when exchanging data even between the NAT-protected private internet peers after having a handshake by means of any middle server?

There is only one way that is using STUN which uses UDP protocol. It requires a STUN server which introduced 2 peers and after that both of their NAT are "punched" and allow direct peer to peer traffic. This totally depends on the type of NAT involved and may not work correctly since a lot of home routers are cheap crap devices.

That is why any system using STUN also ends up with TURN server. A TURN server is used when STUN fails to connect. With TURN, both the peers use the server at the middle man to exchange data.

This is super unreliable and requires running server infrastructure and thus it was not used with Mesh. Mesh on the other hand uses a TCP based protocol and uses DHT network. If Mesh gets popular then there will be no issue with connectivity for anyone since there would always be some node online to connect to.

shubhamdhanetia commented 3 years ago

Thanks for clearing my concepts! It's really improving my Computer network knowledge in a more practical way.

So to summarize,

Case 1: Both peers are behind the NAT firewall

Case 2: One peer is using public IP address and the other behind a NAT firewall

ShreyasZare commented 3 years ago

Thanks for clearing my concepts! It's really improving my Computer network knowledge in a more practical way.

You're welcome.

So to summarize,

Case 1: Both peers are behind the NAT firewall

  • Will exchange data using a tunnel server (public node)

  • loss of bandwidth for the tunnel server

Its not a "server" in typical sense. Its just another Mesh node running. Its just that you are trying to run a node on a server just to make the Mesh network stable.

  • If a peer sends a big file to another peer, will it create a bandwidth load on that single tunnel server or it can divide the file into chunks and uses multiple tunnels to send the file?

All data be it file or text chats will be routed via the tunnel so the node being used for tunnel will have its bandwidth consumed. File is downloaded as a single stream only, no torrent like implementation.

Case 2: One peer is using public IP address and the other behind a NAT firewall

  • Direct connection can be established.

  • Won't use tunnel server, so no extra bandwidth loss

This is right. One peer on the DHT network with public IP address will also automatically allow any other node to use it as a tunnel to help them connect with each other.

shubhamdhanetia commented 3 years ago

All cleared ! Thank you very much :)

I was thinking may be free services like ngrok could help when there is no node on DHT network. ngrok provides easy API to create tunnel to the local machine port.

Do you think if it is possible this way ?

https://ngrok.com/docs

ShreyasZare commented 3 years ago

I was thinking may be free services like ngrok could help when there is no node on DHT network. ngrok provides easy API to create tunnel to the local machine port.

  • If the node is behind NAT and there is no current node on DHT network to help, using this tool 'Ngrok', we can create free tunnel to our http or tcp port. It will then generate a public endpoint which will do port forwarding to our local machine. This way, peer outsider our network would able to connect to us using this endpoint and there would be no firewall issue.

  • We can maintain dictionary with key as userid and value as public endpoint generated by ngrok. So, if I try to connect to a peer which is also behind NAT and there is no node available to help, we will call mesh.im service to check if ngrok tunnel available for the userid that we want to connect.

  • Whenever we login to mesh, it should automatically creates a tunnel and registered generated ngrok endpoint(tunnel) in mesh.im service if it's behind NAT.

Do you think if it is possible this way ?

https://ngrok.com/docs

If a 3rd party service is integrated with Mesh then it no longer remains a p2p messenger and its an unwanted dependency too. Making a messenger that works with servers is the easiest thing to implement.

This project aims to be 100% p2p, even mesh.im running a connectivity check service which doubles as a "tracker" is what I would like to remove if there is a way to do it.

Better solution is to just pay a small fee to your ISP to get a public IP address (I pay around INR 100/month). Some ISPs give public IP address to all customers for free and if its not the case then many give it free when you ask for it. Its quite simple to use too, just enable UPnP on your router with public IP address and you are all set.

shubhamdhanetia commented 3 years ago

Yeah, Agreed. Sure, I will ask my ISP regarding public IP Address. Actually I mentioned the use of third party service for the worst case scenario when no public node is available to help, just to make the application work for anyone !

Tycho-X commented 3 years ago

To do that, there is a simple API hosted on mesh.im that provides connectivity check (to let the app know its external end point and to test if inbound connections work) and since all nodes do connectivity check, the API also returns a list of last known nodes.

Isnt this a blatant security hole?

ShreyasZare commented 3 years ago

To do that, there is a simple API hosted on mesh.im that provides connectivity check (to let the app know its external end point and to test if inbound connections work) and since all nodes do connectivity check, the API also returns a list of last known nodes.

Isnt this a blatant security hole?

How?

Tycho-X commented 3 years ago

To do that, there is a simple API hosted on mesh.im that provides connectivity check (to let the app know its external end point and to test if inbound connections work) and since all nodes do connectivity check, the API also returns a list of last known nodes.

Isnt this a blatant security hole?

How?

IF i did understood this correctly it means that the clients will establish a non "obscured" connection to one of your API servers. Hence, it would be possible to get the IP addresses of all clients for someone who can trace traffic to your servers?!

ShreyasZare commented 3 years ago

Well, anyone can use the api to get a list of IP addresses of few other mesh users. This is similar to how BitTorrent trackers work to allow discovering other peers on the network.

This is required since without knowing a handful of peers its not possible to join a peer-to-peer network.

Once a user joins the Mesh Distributed Hash Table (DHT) network, the protocol requires exchanging more peer addresses to be able to access the network.

This is thus a requirement to know IP addresses of nodes on the peer-to-peer network without which its not possible to run a peer-to-peer protocol.

There are no security implications of knowing IP addresses of other Mesh nodes.

One could argue that privacy is at stake but this is what a peer-to-peer network is. Its either you use a centralized network which collects all user data or you use a peer-to-peer network which discloses IP addresses of few random peers and no one entity has access to all the data.

There is one option with Mesh to use anonymous profiles that use Tor network to prevent disclosing your IP address. But this feature works only when there are at least a couple of nodes on the peer-to-peer DHT network.

Shreyas Zare Technitium

On Sun, Oct 3, 2021, 18:06 Tycho-X @.***> wrote:

To do that, there is a simple API hosted on mesh.im that provides connectivity check (to let the app know its external end point and to test if inbound connections work) and since all nodes do connectivity check, the API also returns a list of last known nodes.

Isnt this a blatant security hole?

How?

IF i did understood this correctly it means that the clients will establish a non "obscured" connection to one of your API servers. Hence, it would be possible to get the IP addresses of all clients for someone who can trace traffic to your servers?!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TechnitiumSoftware/Mesh/issues/6#issuecomment-932942168, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACMLM3CIIKUHJNXAET33JCLUFBE3DANCNFSM4VQ6MYMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.