search

swadeshi / wheel

The read/write client for a distributed peer-to-peer network of (social) data
60 stars 0 forks source link

History, Context, Existing tools/projects #1

Open flyingzumwalt opened 8 years ago

flyingzumwalt commented 8 years ago

There are already many projects that have attempted to create a distributed peer-to-peer social network, for example diaspora* and RetroShare. There's an awesome Wiki page with a Comparison of software and protocols for distributed social networking

We shouldn't re-create the wheel if it's not necessary.

Thanks to buovjaga on hackernews who provided some of this info

Do you know of any others? If so, please add comments here.

anotherstarburst commented 8 years ago

The impetus of this project seems similar in theme to what has been discussed around bitcloud (http://www.wired.co.uk/news/archive/2014-01/27/bitcloud) - which aims to take ISP's out of the equation.

flyingzumwalt commented 8 years ago

@anotherstarburst yes. There's definitely synergy between dat/hypercore, ipfs and bitcloud. Thanks for the link!

I think of those as being underlying technologies -- they're not specifically aimed at social data. Instead they're trying to fundamentally re-structure the web of data so that we can do things like reclaim our social data.

geoah commented 8 years ago

I have been interested in something like this for many years, have been through some design iterations (latest one was ink protocol that is now dead) but I've also concluded that federation isn't the way to go so trying to move away from that.

I drafted a request for comments some time ago to get some feedback from friends etc (Nimona RFC gist). The original idea is a GPG (or GPG-like) identity, DHT for identity and node discovery, users can have multiple nodes that coordinate themselves via raft and the leader handles propagating shared data to the outside, communication between nodes is event driven so it makes it easier to sync using a log. Still having some issues with how to deal with schemas and authentication (really not sure about oauth2). - I would really love to hear your thoughts on this rfc btw.

Back to the original topic now, similar projects (or ones that could evolve to something similar) that I know of...

rikmeijer commented 8 years ago

Interesting project. How does it relate to synereo?

flyingzumwalt commented 8 years ago

Thanks everyone for your comments so far. I'm still absorbing all of the comments from hackernews and Reddit -- people made a lot of good points. Even the criticisms tend to be pretty encouraging.

I'll be posting more info soon.

@rikmeijer synereo looks pretty cool. I hadn't heard of it, need to look further. @geoah thanks for all that info. I plan to check out all the stuff you've linked to when I get a chance.

rikmeijer commented 8 years ago

@flyingzumwalt apparantly they havent released anything yet (was unable to find a download link). Also, have you looked at ipfs? https://ipfs.io/

flyingzumwalt commented 8 years ago

Yep. Definitely aware of ipfs. I should have mentioned it in the article. I thought it wasn't exactly suited to this application, but I was wrong. See this twitter conversation with Juan Benet (the creator of ipfs).

I've started putting recommended reading in Issue #3, including links to useful info about ipfs.

daveloyall commented 8 years ago

@geoah I have not concluded that federation isn't the way to go. Email is a very successful federation. What am I missing?

geoah commented 8 years ago

@daveloyall IMO federation has only one advantage... it is user friendly. I can easily remember user@gmail.com, company.com/employee, or student.uni.edu. But what happens when I wanna change providers from gmail to yahoo? We could allow custom hostnames, html metas, dns txt records but in any case we need some for of identification that does not include the provider's name/host/etc. But even then, what happens if I lose control of my domain?

Any way I think of this I end up identifying users using their keys and signing everything with it. Using a decentralized network and a dht or something similar a user can have more than one instances of their servers, they can go up and down and no-one would care. I could add my key to my work identity server, my school identity server, and have one running on my home nas. They'll figure out themselves how to sync the data etc.

The problem with non-federated protocols is that no one really remembers who 3d65059f99b858b2e687a3eb104b6911935ae97186c37041db2706c3ac717ac0 (an ecdsa public key fingerprint for example) is. And there is no way for someone to verify that this is my key. That's where web of trust etc come into play in addition with (as before) dns txt records, metas, etc.

Not sure if that makes sense for anyone else. I understand that a decentralized system has a lot more issues to solve, but email, xmpp, diaspora seem very restrictive.

@daveloyall could you please give your reasons for a federated network? I would be really interested in some fresh ideas on this :D

daveloyall commented 8 years ago

@geoah Wow, this makes me feel old... :)

In the old days*, every host (device) had an IP address and hopefully a memorable name. Some examples:

Email is called a federated protocol, but there was nothing "federal" or centralized about it. It was just a program that runs on your computer and talks to programs running on other computers.

Let's say you want to email your buddy Random J. Hacker.

TO: rjh@athena.ualr.edu
FROM: you@hubris.the-good-guys.net
SUBJECT: AI winter is coming
lorem ipsum sum

.

A naive mail program could just open a TCP connection to athena.ualr.edu on port 25 and say HELO. Notice that I left out anything about MX servers, etc.

A new machine could join the federation just by acting like a mail server. That is, listening on port 25, responding to HELO, etc.

This enabled the proliferation of lots of different types of mail software. But, they all spoke a common language. They were a federation. It worked. And to this day, you can still get email from, for example, princes in Nigeria.

Email is a global communication network capable of sending arbitrary content straight to any specific computer in the world--except that's not how it works anymore. ISP packet filtering of all port 25 traffic, NAT routers, neutered operating systems, @gmail.com (where they stuck all the users on one machine for some nefarious reason), etc have all broken the dream.

* but not TOO old... We could talk about the postal mail system, which also works world wide, if you like! :)

flyingzumwalt commented 8 years ago

Does anyone have a good definition of federated vs. distributed? There's Paul Baran's distinction between Centralized, Decentralized and Distributed (1967. See http://www.decentralizedweb.net/learn-more/) and I've come across lots of things that use the term federated, but I'm not clear on what federated actually means. Based on some descriptions of federation, it seems like distributed and federated are not mutually exclusive. It seems like you could potentially have have systems that are distributed in structure (using Baran's meaning of distributed) but still sometimes use patterns of federation when machines exchange information. Can anyone offer some info to clarify?

daveloyall commented 8 years ago

I did some web searches just now and found out that I might be using an uncommon (incorrect?!) definition of "federation".

There are other people on the internet calling email a federated protocol, however, they are doing so within the context of discussions about google Wave. I suddenly recall that I'd never heard the term "federated protocol" until I became a Wave fanboy myself... Thus... Maybe google was stretching the term a bit.

I apologize if I've perpetuated a shoddy definition!

flyingzumwalt commented 8 years ago

@daveloyall it also came up in the comments on HackerNews so you're not alone. I was hoping for clarification of what they mean by federated.

geoah commented 8 years ago

This is my personal understanding of federated/p2p/decentralized networks.

Let's start with Wikipedia's description of a Federated Identity as an example.

In information technology (IT), federated identity management (FIdM) amounts to having a common set of policies, practices and protocols in place to manage the identity and trust into IT users and devices across organizations.

Federation is enabled through the use of open industry standards and/or openly published specifications, such that multiple parties can achieve interoperability for common use-cases. Typical use-cases involve things such as cross-domain, web-based single sign-on, cross-domain user account provisioning, cross-domain entitlement management and cross-domain user attribute exchange.

Basically when you sign up on Facebook, Google+, you can use your identity to login to other places but your are bound to that provider. Same goes with email, diaspora, xmpp, etc, you select a server or pod that hosts your data and then you can talk to people from other servers or pods but when you decide to move to a new provider you will need to create a new identity there and move your data. Each of these providers use different technologies and have different features but allow you to do some basic interactions with others. At some point one of these providers will stop being nice and remove this functionality the way Google did with xmpp from hangouts.

On the other hand with peer to peer and decentralized the provider is transparent to the network and nodes. They don't care where who the provider is, they only have a identifier that is global to the network and they look it up on a DHT, blockchain or other type of registry, usually by asking the nodes they now of. If a user changes providers nothing bad happens. On p2p/decentralized everyone can be their own provider, you don't even need the notion of one, every node is equal.

In conclusion I think this is the reason why not many people use the term "federated" to describe protocols but mainly use it for services; and it is control. On a federated service you might still not have true access to your data.

@daveloyall @flyingzumwalt sorry for the rant and hijacking of this thread! :)

daveloyall commented 8 years ago

@geoah Well, let's not conflate the behavior of megacorps with the nature of the tech.

I think "federated" protocols like OAuth, xmpp, email, etc are solid theoretically and they don't favor megacorps or deny users freedom.

The issue is that the majority of users (even "us") didn't own the nodes. If you only ever spoke XMPP via Google's servers, then you were never a first class citizen on/in the XMPP federation.

Google's chat service could be thought of as a single huge multi-user computer. When they changed the software on that node, lots of people lost presence on XMPP (I guess? I'm not up on this history). But XMPP didn't go away, the design didn't change. The admin on that big computer just uninstalled some software. If we'd all be running our own nodes, we'd not have cared.

OAuth went wrong in two ways: first, just like XMPP (and email), hardly anybody ran their own nodes. We used 3rd parties (Google+, Facebook, etc). The second, much worse failure was that many sites that aimed to support OAuth ended up supporting Google+ and/or Facebook specifically rather than the protocol generically. I was promised that OAuth would let me log into my bank, my gmail, and github all using my own me@the-good-guys.net identity. I never got my "node" up and running, but even if I had, I can't remember the last time I saw a site that offered to let me sign in using an arbitrary domain. [Edit: I think I confused OAuth and OpenID here... See these diagrams.]

All that being said, I'm surely happy for Wheel to utilize a DHT or blockchain or etc. I don't know much about that stuff but I hear it is great. I think these technologies are necessary to provide "hosting" of content in absence of personal servers (personal devices that respond to requests from the net).

BUT, (and this is why I haven't relinquished the mic just yet... :) ) ... if the app that grandma uses to read or write content to the DHT lives on some 3rd party's server, she still won't be free. Because if that provider is @gmail.com, well, they could drop that feature whenever they want. So merely utilizing some fancy tech won't free users, there has to be a sort of social shift.

We can't make an App. We have to make a protocol, so that any app that speaks that protocol can participate. Like CB radio vs cell phones. Use of the former is free; user of the later is, well, rented!

geoah commented 8 years ago

@daveloyall can we imagine a distributed protocol (we'll call it xyz for now) where grandma registers in an online service (awesome-provider-1) that supports our xyz protocol and starts using it. She adds her grandsons and her daughter and can now use any application that supports our xyz protocol. At some point when registering on the provider, grandma received her own private key that will be told "not to lose" (this will be a nightmare in the long run but that's a different issue for another time.) which we will assume that she actually keeps safe.

At some point one of her grandsons buys her a cheap storage device that is always on and has support for our xyz protocol, plugs it in her router and grandma adds her private key. In a couple of hours or days, the router will be in sync with the data in awesome-provider-1, now even if the provider goes down or decides to shut down, grandma's data are safe and she can still use her apps since xyz is a properly decentralised protocol and the apps will figure out they should be now talking with her nas.

At some point grandma will phone her daughter to complain about her photographs taking too long to appear and they will sign grandma to a paid provider this time just by registering, paying for the disk space and adding her private key, eventually everything will be in sync and all is fine.

ps. The grandsons are running client/servers of xyz that only syncs pictures and videos on their mobile phones since they are really scared of losing their photographs and have like 1Tb sd cards on them.

daveloyall commented 8 years ago

That sounds great. Well, one nitpick: I don't like the part about grandma sending her private key to the paid provider.

Granting a third party access to data without giving away the private key has got to be a known use case in the PKI world. What solutions have been identified already? Maybe the provider generates a private key for her on-site and she subsequently signs it? I dunno, I'm no PKI expert.

gsf commented 7 years ago

Just read your article and came here to see if IPFS had come up on the radar. Glad to see it has because that seems to be where the momentum is right now. I was running the client on a server for a while but had to stop it because of memory and network usage -- I think it will really catch on when it can be run anywhere (including mobile) without much of a footprint.