search

ipfs / notes

IPFS Collaborative Notebook for Research
MIT License
402 stars 31 forks source link

Matrix #42

Open davidar opened 9 years ago

davidar commented 9 years ago

Several people from both the @ipfs and @matrix-org communities have expressed an interest in the possibility of integration between the two. I'm opening this issue to coordinate this effort.

Relevant links:

CC: @jbenet @ara4n @dbkr matrix-org/matrix-spec-proposals#33

hackervera commented 9 years ago

:+1:

jbenet commented 9 years ago

For starters, we should:

ara4n commented 9 years ago

On the Matrix side, we've been looking at this on three levels:

  1. hosting matrix web clients in IPFS (which @davidar looks to be doing a great job at with Vector)
  2. potentially using IPFS as a means for storing and sharing files in Matrix. Currently we have a naive HTTP-based replication strategy between servers; IPFS might be a huge win, with the IPFS node itself either sitting in the client, homeserver or IPFS gateway (in descending order of preference).
  3. the thought experiment of using IPFS as an alternative server<->server transport/protocol for Matrix.

The 3rd point is the most controversial, as Matrix and IPFS's goals start to overlap somewhat at this point in that they both want to provide a big global DAG; Matrix currently optimised for pubsub group messaging; IPFS currently optimised for files. Right now the server-to-server protocol for Matrix is core and pretty vital for us to get perfect, and we're expecting to keep using it for our state synchronisation and pubsub for the forseeable rather than making it dependent on any 3rd party project.

However, it is also designed to be replaceable - so down the line, one could certainly consider replacing it with something like IPFS (or a Tor + Pond style transport). We'd go and bridge the current Matrix network into this alternative one, and then on with the show. It would be a very significant amount of work to replace the replication of a matrix homeserver, though, and to map the requires permission and merge resolution semantics onto a hypothetical IPFS pubsub system.

The fun thing about Matrix is that even if the federation transport is completely replaced in this manner, all of the current Matrix clients, bots and application services (e.g. bridges) would be untouched and still work absolutely fine. So if someone ever did make Matrix homeservers store/synchronise their data via IPFS, you'd be able to use it seamlessly with all the existing apps :)

In terms of storing logs in IPFS... one could certainly write a Matrix AS that copies Matrix events into IPFS as a 'log', and then write a log-viewer app to view them. However, we'd obviously hope that the Matrix DAG itself performs this job pretty well now, and you don't need a separate log-viewer app as you can just go and use any old Matrix client to view the Matrix DAG. That said, nothing to stop someone going and doing this anyway if they want to :)

ara4n commented 9 years ago

oh, on the whole "one-click-to-join-chatroom" thing... hell yeah, as per our discussion at the PDX meetup. We should have this finally implemented in the next month or so, with any luck.

jbenet commented 9 years ago

@ara4n awesome stuff all around.

one quick note for now: we've had a goal of building pubsub. we'll get to designing this pretty soon. if you help us understand your requirements, we can make sure to meet them. (or even use protocols from matrix if you're ok with that). it may be that what we should be doing is using the matrix pubsub mechanisms to move ipfs merkledags inside ipfs itself. i'll look at your specs in next week(s)

ara4n commented 9 years ago

Most of the matrix spec is concerned with the client<->server and application<->server APIs rather than server<->server (aka federation) API. The best we have right now on the smart stuff (i.e. ACL-aware merge resolution algorithm) is https://github.com/matrix-org/matrix-doc/blob/master/drafts/erikj_federation.rst.

The hard thing here isn't the pubsub, which can just be a simple "prod listeners when stuff changes", but having the right semantics for propagating updates and solving merge conflicts on said stuff in the context of the room. Needless to say, if you want to experiment with applying Matrix's semantics on top of IPFS, we're not going to stop you - and as per above you'd automagically gain a bunch of clients/bots/etc by doing so. But it could be a large amount of work. It may get easier over the next few months though as we go through splitting out our homeserver implementation into separate services - some of which will probably be written in Go rather than Python. One of those services is likely to be the federation service, so if we structure things right we might actually get a modular system for swapping out federation transports for this kind of hackery anyway!

jbenet commented 9 years ago

nice! this is great to hear. will look deeper into this stuff as time permits. if anyone else beats me to it, thank you!

The hard thing here isn't the pubsub,

i'm worried about making it scalable in light of massive churn (Millions+ nodes). IPFS nodes will be coming up and down all the time.

Ericson2314 commented 5 years ago

I'm eyeing https://github.com/libp2p/rust-libp2p, https://github.com/ruma/ruma, and wishing.

swedneck commented 5 years ago

Me and tulir discussed how IPFS could be used in the matrix media repo, and how mxc:// urls could be decoded into IPFS hashes: https://github.com/turt2live/matrix-media-repo/issues/115

I've also published Riot on IPFS: https://riot.swedneck.xyz

bmiller59 commented 4 years ago

Hi @ara4n. Are there any update on using IPFS to store files for Matrix worth mentioning here? If someone wanted that to move forward, what would you recommend as some steps towards that goal? Thanks! Referencing this here as well: https://github.com/matrix-org/matrix-doc/issues/539

ara4n commented 4 years ago

well, https://github.com/turt2live/matrix-media-repo/issues/115 moved forwards last week...

blaggacao commented 4 years ago

It would be awesome to amalgamate matrix + ipfs into a full collaboration toolbox, lending ideas from kbfs and giving matrix-pub/sub-honor to this innocent question at recent talk minute https://youtu.be/Z5zNPwMDYGg?t=3435 about answers to the shifting CID for references to mutated data. Matrix could leverage ipfs's foundations and present an interface to implement a unified, shared, zero-trust, yet team-encrypted fuse filesystem, while solving the shifting CID issue through the matrix protocol (in the corona-pandemic collaboration context).

Use case & plea

blaggacao commented 4 years ago

@ara4n You've probably seen this already, which got recently merged. But I just made the connection with your presentation of libp2p and other p2p experiments on this year's fosdem.

RubenKelevra commented 3 years ago

Has there been any progress using IPFS as transport for Matrix? Would be absolutely awesome if you could use @nick:domain.org and just add a text record on the domain to specify the ipns. Like nick.ipfs.matrix.domain.org.

The ipns would need to point to a profile folder which contains the individual clients as text files with the ipfs public key and the matrix public key for encryption as well as profile data like the profile picture.

The connections would need to be established to the clients directly instead of a centralized server. Easiest solution would be to create the server portion directly in the clients. Which means a foreign server would connect to the clients in the ipfs folder following a prioritization.

The individual clients would need to connect between each other to exchange notifications about new messages, while the messages could be stored in ipfs directly.

After like 14 days individual messages would need to be compacted in single flies for each chat, to avoid the overhead of hundreds of thousands single message files 😁

Ericson2314 commented 3 years ago

See

You will notice that the blog post was a bit negative on libp2p's current methods and was looking at Yggdrasil. But that would be a shame, as it's great how so much of the ecosystem has been moving towards libp2p, and since libp2p aims to be "mechanism not policy" (it's a library of things one could use). Hopefully Yggdrasil-like spanning tree mechanisms could also have a home in libp2p so there's not a collaboration/experimentation trade-off.

Ericson2314 commented 3 years ago

At the risk of being the annoying fan shipper here :D, https://matrix.org/blog/2021/05/06/introducing-the-pinecone-overlay-network has an update saying:

If Pinecone works out, our intention is to collaborate with the libp2p and IPFS team to incorporate Pinecone routing into libp2p (if they'll have us!) while incorporating their gossipsub routing to improve Matrix federation... and get the best of both worlds :)

Which is just the sort of thing I am hoping for! I eagerly opened https://github.com/matrix-org/pinecone/issues/7 in response to track any such developments.