automerge / hypermerge

Build p2p collaborative applications without any server infrastructure in Node.js
MIT License
1.28k stars 66 forks source link

[WIP] Feed Store, Network, and File Server #31

Closed mjtognetti closed 4 years ago

mjtognetti commented 4 years ago

Extracts feed management and network management into separate modules so feeds can now be replicated without actor involvement. We need to avoid involving actors because actors load entire feeds into memory. Having a separate replication layer is a first step towards efficient replication of all documents and feeds - which ultimately enables cloud peers and more effective/reliable syncing.

Also introduces an HTTP file server for reading and writing hyperfiles. This solves our bus problem of intermixing binary data with JSON messages. The FileServer is intended to be used with an IPC connection (unix domain sockets on unix, named pipes on windows), avoiding network overhead. Additionally, hyperfiles are built entirely on streams, enabling more efficient reading and writing.

TODO:

Future Work:

mjtognetti commented 4 years ago

Will probably re-up this PR without dist changes.