search

HelloZeroNet / ZeroNet

ZeroNet - Decentralized websites using Bitcoin crypto and BitTorrent network
https://zeronet.io
Other
18.39k stars 2.27k forks source link

DHT support #57

Open HelloZeroNet opened 9 years ago

HelloZeroNet commented 9 years ago

Similar to torrent's DHT solution, required features:

Maybe its requred to separate the DHT from Peer Exchange

Please comment if you have other ideas/suggestions.

defnax commented 9 years ago

why not use libtorrent?, twister uses libtorrent for DHT http://twister.net.co/?page_id=54

HelloZeroNet commented 9 years ago

zeronet protocol is different from torrent, so libtorrent will not work. Also bittorrent DHT is UDP based, so it will not work on Tor without proxying it to TCP which I don't want. And it does not support storing Tor hidden service addresses, so you would not be able to create sites on Tor.

TheNain38 commented 9 years ago

@HelloZeroNet You could add to your first comment:

HelloZeroNet commented 9 years ago

Its already there:

TheNain38 commented 9 years ago

@HelloZeroNet Ho, didn't see that, sorry

alxbob commented 9 years ago

https://github.com/Ayms/node-Tor

https://pypi.python.org/pypi/pyp2p

alxbob commented 9 years ago

Full tor support and hidden services can work inside zeronet by creating a dht (that would be a tor hidden service it self) that holds the .onion addresses and load balances them. In clearnet it could work as a common dht. So its users connecting to zeronet through tor will have a hidden service created for its site he/she visits. And let tor hdirs take care the rest!

https://pypi.python.org/pypi/OnionBalance

https://stem.torproject.org/tutorials/over_the_river.html

pldubouilh commented 8 years ago

Interesting issue. Zeronet would drastically benefit from a DHT, as trackers tend to come and go with time...

My bet is that it would be best to bootstrap and use Bittorrent's DHT (Mainline DHT - MLDHT), as although it has a few problems, a lot of effort is put into it to keep it reliable and stable. A simple approach could be done using announce_peer and get_peers. Some more fancy stuff could be done using the post-signed-mutable-content thing (see bep_0044), but that might not be interesting for Zeronet.

For instance I (new user) query the MLDHT : get_peers(sha1(1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D)) and retrieve a list of the peers on it - simple. The peers would have to first announced themselves using (roughly) announce_peer(sha1(1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D), my_ip_and_port)

About the dependency, I think it's inevitable, at least for some things like bencode and the MLDHT, although there seems to exists relatively small python libraries. I'll have a look at this.

SeniorPlayer commented 8 years ago

Can the DHT solves the problem that how a new client first join the zeronet when the tracker server is down?

HelloZeroNet commented 8 years ago

@stillwarter DHT still need a bootstrap server, so it does not help on new "virgin" clients.

obv-mikhail commented 8 years ago

@HelloZeroNet Maybe this https://github.com/maidsafe/routing or this https://github.com/maidsafe/crust can be used.

alugarius commented 8 years ago

Custom bootstrap clients (storing all of them is nice too) , some proxys and an option to save an anonymous list of users to use as emergency bootstrap, it can be stored(exported and imported)! The "unvirginizer" ( IPFS tactics)

Bachstelze commented 8 years ago

Could we use https://github.com/closeio/redis-hashring in Python or https://github.com/RJ/ketama in C?

sim590 commented 7 years ago

Hi,

I propose to you to consider using OpenDHT. It has the following features:

I personally would like more projects to join like Syncthing.

slothbag commented 7 years ago

IPv6 for CJDNS would be fantastic.

MuxZeroNet commented 7 years ago

I talked with Shortcutme with this issue. Here are the questions he is trying to find an answer to.

Some months ago I did some experiments and checked out some [DHT] libraries, but there are still lots of questions left.

Here are my questions:

d-roak commented 7 years ago

Which DHT protocol do you use ? @HelloZeroNet

alugarius commented 7 years ago

Does ZNet made any progress in this issue?

HelloZeroNet commented 7 years ago

New progress made yet. Does anyone know already working P2P application that uses DHT over Tor? (preferably with >1000 nodes)

skwerlman commented 7 years ago

this might be of interest. It's not reimplementing DHT using TCP, but instead it just carries the UDP traffic through a TCP tunnel (with the express intent of letting it be carried by Tor)

gitbugged commented 7 years ago

Very interested in getting DHT, but in the interim with trackers getting blocked and dropped, maybe you could add more trackers? https://github.com/ngosang/trackerslist

MuxZeroNet commented 7 years ago

Here is my own implementation of the Pastry DHT algorithm. https://github.com/MuxZeroNet/pastry

Pastry basics:

Joining, leaving and repairing the network:

The packets you may need to provide:

  1. A packet that implements this interface: has_value, other_nodes = send_dht_request(peer, key)
  2. A "gossip" packet that allows nodes to exchange parts of the routing table or leaf set.

Edit: Removed "Toy DHT." Added Pastry, which is not a toy.

MuxZeroNet commented 6 years ago

We got a new problem:

Tamas Kocsis wrote: The handshake and the encryption has pretty big overhead so probably it would be more efficient if we would use a separate UDP port

Yes. In practice, latency is what makes DHT protocols slow, which is one of the reasons DHT protocols run over datagram, but rarely over TCP streams.

In Tor, you have to use TCP streams, because this is how Tor works.

I2P SAM provides datagram API. Running DHT over I2P has "encryption overhead" and has observable latency, but it is better than nothing. DHT has been used by I2PSnark and BiglyBT for torrents.

Do you think if the DHT should be part of ZeroNet protocol? [use the protocol provided by] one of the already existent [libraries]. (maybe libp2p or any other dht lib)

I am more on the "implement a DHT that fits ZeroNet's needs" side, while I am not familiar with the API libp2p or any DHT library provides.

Even if you will be using a DHT library, you should always think about how to write the protocol docs. By using proper means of abstraction, the less you leak implementation details in the specification, the more flexible the protocol is. You really don't want to make future developers who will be implementing ZeroNet in another programming language get stuck on a particular library.

HulaHoopWhonix commented 6 years ago

For a DHT alternative you might want to look at how syncthing implements TCP relays to allow hole punching and communication over Tor. Magic-Wormhole a secure simple file transfer tool is looking to adopt their design to solve scalability.

https://docs.syncthing.net/users/strelaysrv.html

list of servers: http://relays.syncthing.net/

cc/ @adrelanos

sim590 commented 6 years ago

@HulaHoopWhonix The thing with relay servers, is that you still have a form of centralization: indexation system of IP addresses. If the indexation system is put down, you cannot query the list of IP addresses. Even worst: censorship, mitm and all centralized scheme attacks are possible. When using DHT, a new routing protocol emerges on top of the IP network which is (really) hard to take down and attack once you have passed a certain point in the number of nodes. After you have resolved the hash query, you can exchange with your peer through DHT put and get operations to go on with hole punching.

However, what you speak of makes me think that onion services may be a good fit for syncthing's use case. Each device could host an onion service in the tor network when it's online so that it can exchange ip address with each other. I'm throwing that idea out there, but I don't know a lot about the onion service design... Also, it seems funny to use an anonymizing network to exchange IP address. May be syncthing could even run directly on tor, but I digress from the original subject.

HulaHoopWhonix commented 6 years ago

Any network, including DHT and P2P ones will rely on bootstrap nodes to help them reach the wider network. This is unavoidable. DHT can never be used over Tor and that's why a commonly emerging solution in the Tox project and Syncthing is to rely on some form of federated relays.

Each device could host an onion service in the tor network when it's online so that it can exchange ip address with each other.

You realize that onions don't have IPs?

While Tor support in the form of ephemeral services using Txtorcon are a neat addition it still doesn't solve the problem of a Tor user interoperating with the outside zeronet network that runs on on the clearnet. Unless Tor use is enforced by default, you will end up with a fragmented network.

sim590 commented 6 years ago

Any network, including DHT and P2P ones will rely on bootstrap nodes to help them reach the wider network.

Yes. However, listing relays on a centralized machine, contrary to using a DHT, doesn't scale.

DHT can never be used over Tor

I didn't say that. However, I assumed the opposite. In fact, onion services use DHT.

that's why a commonly emerging solution in the Tox project and Syncthing is to rely on some form of federated relays.

As I acknowledged above, bootstrap nodes are needed at first, but you don't need to solely rely on fixed relays over time. Once you're connected to the network, to a DHT for example, you save that list of ip addresses you have contacted, you don't rely on the centralized ip address list and you continuously learn about new nodes through the protocol itself, hence the scaling capability. Opting for centralized relay indexation system implies lacking of scale or reinventing the wheel, like the DHT that is. I'm sure that you know that.

While Tor support in the form of ephemeral services using Txtorcon are a neat addition it still doesn't solve the problem of a Tor user interoperating with the outside zeronet network that runs on on the clearnet. Unless Tor use is enforced by default, you will end up with a fragmented network.

Excuse me. I recognize(d) that I digressed from the subject in the last paragraph of my last post. I was not speaking of a solution for ZeroNet.

You realize that onions don't have IPs?

What exacly do you imply? Please, can you avoid making sentences in the form of "You realize that [place some supposed well-known fact here]?" They're not clear and tend to be taken with offense.

I think I understand what's the concern here. You're thinking that I'm speaking of onion services for ZeroNet, but I'm not. I'm speaking about the case of Syncthing. Onion services are resolved with a distributed hash table lookup if that's what you meant by your sentense above. What I meant is that Syncthing devices could use onion services to exchange their ip address like you'd do in a distributed hash table, except that they'd do it through the tor network in addition to looking up the DHT. That may not be adequate indeed after a second thought. A DHT is sufficient, I guess.

I think that we should avoid continuing to speak about Syncthing now as it is not the concern of this issue.

weimilianqiao commented 6 years ago

@HelloZeroNet

New progress made yet. Does anyone know already working P2P application that uses DHT over Tor? (preferably with >1000 nodes)

Please consider retro share. Dark net mode of Retro share let tor or i2p do the peer discovery. It is NOT DHT over TOR. http://retroshare.readthedocs.io/en/latest/user-guide/settings/#network

Dark net mode is used if neither DHT nor Discovery is wanted. Tor and I2P hidden nodes use this by default as they already have a fixed address to connect to.

vRobM commented 6 years ago

https://github.com/HelloZeroNet/ZeroNet/issues/57#issuecomment-320295846 Definitely Kademlia. https://github.com/bmuller/kademlia @HelloZeroNet

HelloZeroNet commented 5 years ago

https://github.com/bmuller/kademlia: "The nodes communicate using RPC over UDP to communiate, meaning that it is capable of working behind a NAT." Tor is TCP-only so it's not going to work

mwarning commented 5 years ago

How about DHT only without Tor for now?

agentofuser commented 5 years ago

@gpestana is very knowledgeable about onion-routing + DHTs, maybe he can help. There are some useful links here:

gpestana commented 5 years ago

hey! ZeroNet is great, good job! 🚀🚀

We at hashmatter are very interested in understanding how different projects are using DHTs and P2P routing protocols and what implication it may have in privacy and metadata leaks - and to try solving privacy vulnerabilities. In the context of ZeroNet, what would be the goal of a DHT? Would it be for service/peer discovery or/and storing and requesting content? How about the latency requirements? In terms of metadata leaks, DHTs may become very problematic and proxying the DHT traffic through Tor may sometimes not be the best solution (depending on the latency requirements, peer identity requirements, etc.) I believe the best way is to define 1) what are the goals for the DHT 2) latency requirements 3) threat model (this might exist somewhere already, and I'm sorry if that's the case and I didn't research enough). There are interesting protocols that can be built on top of the DHT routing protocol (check some here https://github.com/hashmatter/p3lib) that can enhance considerably the privacy of DHT interactions without the need for routing all traffic over Tor.

Happy to keep the discussion and helping if needed 👍

HulaHoopWhonix commented 5 years ago

@gpestana I'm sorry but nothing you can come up with can match or surpass the anonymity of Tor which has been around forever and has a proven track-record.

Unless your solution is compatible with Tor, it is not relevant for the goal of this ticket.

erik777 commented 5 years ago

The red herring seems to be the requirement:

It should work using on Tor (TCP only, maybe UDP on clearnet)

Can we generalize this requirement? What are the requirement for Tor? Anonymity? Privacy? So, how can you use DHT in a way that preserves these basic requirements, while not necessarily using Tor and not impeding the ability to use Tor for the rest of ZeroNet's functions?

So, when I look at OpenDHT, I wonder to what extent this capability helps meet these core requirements:

Public key cryptography layer providing optional data signature and encryption (using GnuTLS)

EternityForest commented 3 years ago

One of OpenDHTs very nice features is a defined REST API for DHT nodes, which act as gateways to look up keys without actually using the DHT yourself. Why not just switch over to using this mode of access when using TOR, but use the DHT as is when using the clear internet?

Trackers can come and go all they want, just switch to another, they all use the same DHT backend.

nerd36 commented 3 years ago

One of my blog (zite) visitor complained about there is no peers error but there was 20 peers. The tracker was lying. I created this bounty. At this point myself cannot pay anything because anonymous gift card is not available in my country Next step could be block-chain bootstrapping

nerd36 commented 2 years ago

it will not work on Tor without proxying it to TCP

You should use Lokinet instead of Tor it is safer because monetized and it support UDP. Even maybe they can sponsor you