Direct ip-2-ip tunnelling of payload data (not relaying it)

[gianni76]

After initial testing of uTox and verifying that in TCP-only mode (UDP off) the bandwith consumption was acceptable, I am happy with the functionalities, but I learnt that in this mode uTox clients don't connect directly one-to-one in p2p mode, but rather via 'other people who are acting as TCP relays'.

From the normal usage needs of lots of field workers, I would like to suggest the implementation of a far more efficient configuration, which, after the initial discovery phase (DHT id-to-ip lookup), sends the payload data (chat/voice comms and/or files) directly via tunnels from-source-ip-to-destination-ip. This would ensure the maximum possible:

• efficiency (no data wastage),
• reliability (if we both are online, after the discovery phase, even if no relays are online, we can still exchange data), and most of all
• security (third parties data tapping/recording would be much, much more difficult).

The security concern is maximum for me, as passing via relays (as it is done now) of course weakens a lot the communication link specially for files and allows easier potential data leaks (yes data would be encrypted, but still an attacker could more easily record them and attempt to decrypt it. Even if it took a long time to do it,... files could be exposed).

Of course relaying could be useful/essential for particular network configurations (NAT-ed links for example), but in 99% of the "normal" cases the ip-p-2-p-direct-tunnelling is the most efficient and secure way forward and should be at least an option in the program.

[gianni76]

I should add that I strongly support as well the request of having a direct-IP bootstrap without being forced to go trough DHTs made here:

https://github.com/irungentoo/toxcore/issues/1453

In fact in some configurations, we have machines on the field which connect for other reasons to DNS-IP-recording services, so someone wanting to connect to that person might simply get the IP from this external mean/tool/have it and connect directly and more securely (less useless DHT lookup activity equal to less traceable user data left)

[jf99]

I generally agree that two friends should connect directly to each other if possible, no matter if they use UDP or TCP. However, I'm unsure how many users would actually benefit from it. At least for non-mobile internet access, NAT and router firewalls seem to be "normal", so at least one of the friends would have to unblock/forward a port in his router.

Note that finding friends over DHT is efficient and scalable ( O(log n) with n being the number of tox users ). In contrast to that, the way most (all?) clients currently do bootstrapping is not scalable (and easy to censor), because the initial connection always goes to one of the hardcoded supernodes. This has already been criticised multiple times but afaik nothing has changed yet.

I'm not sure whether toxcore or the clients are responsible for this. A good way of bootstrapping would be, if Tox remembered the nodes it was last connected to and tried to contact them first. Also the possibility to have a user-defined list of bootstrap nodes would be great. Note that the ip address is not enough of information. You need the public key of the node, as well.

[LuccoJ]

@jf99 The thing is that UDP is more effective when you want to punch through NAT. If you can't manage to punch through NAT at all, then at that point you need to use relays, and TCP becomes easier. That's why UDP is used for direct connections, and TCP for relayed one.

What you say about boostrapping is reasonable, but I think it's mainly a client issue: it's clients that send toxcore one or more requests to bootstrap, indicating the node(s) to do it from. I'm just unsure about whether toxcore can provide clients with information about nodes connected to after bootstrapping, which would be required for the clients to be able to cache those for future bootstraps.

[ProMcTagonist]

Clients only use bootstrap nodes for the first connection. After peers are discovered Tox saves them and tries to bootstrap off of them next time first before using the hardcoded supernodes.

[jf99]

@LuccoJ I know UDP is used for hole-punching, but the mobile clients use TCP mode anyway, because it causes less traffic (and needs less power). Nevertheless, a direct connection to friends would be beneficial for them as well as for desktop users.

@ProMcTagonist That's new for me. Does this apply to all clients or just some? More specifically, does it apply to µTox?

[ProMcTagonist]

@jf99 I don't use utox ask @GrayHatter

[LuccoJ]

@jf99 but the reason TCP mode causes less traffic and needs less power isn't because "it runs on TCP instead of UDP", but simply because it uses relays and doesn't partake in the DHT in the first place. There is no intrinsic reason why, as a protocol, TCP should be more lightweight than UDP - rather vice versa.

Which is why I don't think it makes sense to say that "two friends should connect directly to each other if possible, no matter if they use UDP or TCP": UDP should be used to connect directly, TCP should be used to connect when relayed. And that's the way it is. Other combinations wouldn't make anything better.

[jf99]

@LuccoJ These are different things:

• Connecting to a node because he's in my friend list and he's currently online.
• Connecting to a node because he's in my friend list and i'm currently chatting/voiping/file sharing with him
• Connecting to a node because I need some peers to do DHT.

For maximum power saving, you would just do to the second - plus one connection to your relay node. I have to admit that I don't know how many DHT peers Tox uses, but it might be more than the average Joe has in his friends list. Anyway, there is potential to keep the number of simultaneous connections low and still use direct connections to friends.

[LuccoJ]

@jf99 for most commonly-used types of DHTs to work efficiently, you need a logarithmic number of connected peers (logarithmic to the total amount of users, that is). I was told at least a hundred connections is normal for other DHTs like Kademlia or BitTorrent's too. But maybe it's possible to make do with less.

Anyway, I wasn't even discussing that: I was simply saying that direct connection is going to work through UDP, while TCP makes sense for relayed connections and not for direct connections. None of this has to do with whatever the client is doing with the DHT.

[gianni76]

Lucco, I am not sure why you continue to insist that the current way (relay instead of direct connections) is 'optimal'.

Separate for a moment the discovery mechanism from the file exchange mechanism. And think to a pure 1-to-1 mode (1 person with a single friend/colleague/supervisor, as happens most of the time).

The transmission part of this mode is in any transmission theory book optimal/less resourse wasting (and more secure as well) when done directly (and not relayed). There is NO DISCUSSION on this. I will not accept any fantasy theory that does not recognise this. And it goes well beyond the particular protocol you use (it is valid for tcp/udp and any other transmission protocol).

So the implementation for mobile (1-to-1) users should allow a way to directly exchange all the data without any relay of any external data (I don't want my data to be relayed out to other super-nodes and I do not want to be a node in a relay/DHT network; so my goal is to create a simple small private transmission channel for colleagues and not share anything externally).

Then, separately and as a totally different problem, you can discuss/think of an optimal way to do the initial peer discovery. The initial discovery could be done with a DHT with voluntary participation (for example those on large networks and excess bandwith) or via intelligent combinations of IPs and hidden services (as richocet is doing without relaying anything or wasting bandwidth).

What I have asked is a way forward that thinks to this serious issue for mobile users asking for private 1-to-1 efficient communications. Of course, if nothing is done, sooner or later Richochet will implement file transfer so most of the mobile users I know will probably move en-mass to that system.

[LuccoJ]

Whatever, really. I've explained things clearly, and that direct connections can and should be made through UDP and not TCP, and I'm not discussing with someone who calls what I say "fantasy theories" and the like. After all, you've said there is "NO DISCUSSION", so I'll respect that.

If you find someone who will listen to you, they will implement what you want (if they understand what it even is); otherwise, you can always submit a patch, I'm sure.

Or your crucial, critical mass of friends will switch from Tox to Ricochet once they pull their act together, whatever happens sooner. I'm not sure you even understand how Tox or Ricochet work, since Ricochet very explicitly uses the Tor network for all communication, implying that everything is relayed through Tor nodes... but whatever.

[aaannndddyyy]

@gianni76: you say ricochet is better in this. Ricochet relays through 6 other nodes, tox by default only through 1 or none. So you want that we use more relays? You can achieve this but proxying fox through for for both nnected parties. So I guess this issue can be closed.

[mrwicks]

I wrote a long post explaining that I couldn't get the clients I had to do direct P2P communications. As it turns out, my OWN client does do this correctly, which is a version of echo_bot (from the wiki) with a skeleton for file reception - using qTox to send the file. So, P2P does indeed work. Perhaps I just got lucky this one time.

It doesn't work with tuntox, ever, it seems - this turns out to be because you need to use the option "-t 0" otherwise it sets up a TCP relay - still it really liked to connect to TCP anyhow.

I share the frustration of gianni76, as this doesn't seem to work with most clients which is confusing. It almost seemed as all connections went through relays.

BTW - (not that I have any standing here... yet) c-toxcore shouldn't have any sleeps in them at all, it should be event driven with epoll. It's fine to have timeouts in the epoll to do maintenance, but it's just odd to have a fixed 50 ms sleep all the time.

Anyhow, neat project, I'm looking forward to working on it and extending it, and killing bugs.