Announcement: Formalizing purpose & name change

telamon commented 4 years ago

Up until now I've had some issues describing this project and not unsurprisingly that complicates naming as well.

Previously I've advertised replic8 as an replication manager, but that is only one third of the scope and it's not the direction I want to take..

Background

I Joined the community at the end of 2018 when I discovered cabal and realized that multifeed introduces a completely new concept to the dat-ecosystem.

As opposed to replicating preshared keys and resources known ahead of time. Multifeed allowed peers to meet on a common topic and exchange public-keys much like a bazaar where people can meet and trade resources.

Now cabal itself used a very straightforward rule of exchange:

Each peer agrees to replicate all feeds equally.

It's a good rule, plain and fair. But it captured my interest and raised the questions:

Can we design complex rules?
Can rules be modularized into pluggable building blocks?
What would the API look like from the application's perspective?
How is resource management affected if we were to allow volatile feeds or selective replication?

The current state of the project

I've been working on answering those questions for half a year now, and my research has boiled down into the following 3 areas:

Exchange protocol
Replication manager
Application stack

When all three work together we get the concept of Application Defined Replication. Which is exposed to developers via the middleware interface.

Simply put, it gives applications fine-grained control over everything that goes on between storage and replication.

New name & new tagline

Bye bye replic8.

Hello Decentstack!

Decentstack is:

A small decentralized application framework

what Decentstack isn't:

It's not a Core storage ^{Bring your own storage(s)}

It's not a Datastructure _{Can be used within a datastructure}

It's not a competitor to Dat SDK _{It's not an sdk or api, it's a way for you to arrange your
application components into a vertical 'stack' promoting a design-pattern.
Could be made part of Dat SDK; who should I talk to?}

It's almost not multifeed _{If you extract replication and exchange from multifeed into a standalone module, then decentstack is what you get}

Do we need it?

Observing cabal development from the side-lines I've found having some separation enables me to contribute to cabal while at the same time being able to re-use the code in my other projects

So far I've been using the terms Application and Infrastructure to decscribe two problem domains.

By categorizing cabal as an Application here are a few issues that I feel belong to the Infrastructure domain and could either be simplified by using the middleware pattern or be moved to Decentstack's backlog.

multifeed#2 API: selective peer replication #2 (Solved) this is what sparked the project
multifeed#27 Live replication does not forward new feeds (Solved) Predecessor to the ReplicationManager
cabal-core#46 Moderation feature: next steps Perfect example of ADR, application defines the logic while 'middleware interface' exposes replication control
cabal-core#56 Don't replicate zero-length feeds Another usecase for an application defined filter
cabal-cli#130 too many simultaneous feeds prevents scaling. This is an infrastructure issue.

Having more software incorporating the middleware interface is a strong incentive to improve the common ground for everyone, or create stand alone modules that existing applications can use with minimal effort.

So what's next?

This is an open invitation to join the effort and a "help wanted" sign.

And since I personally only extend trust when given transparency, here's my current backlog:

[x] Finish hyperprotocol-v7 upgrade (almost done!)
[x] Create issue: Replication Queue (Plan & Discussion)
[x] Create issue: Missing docs and guides
[ ] Create issue: Feature; how do you want to use authenticated channels? (Plan & Discussion)
[ ] Create issue: Feature; virtual streams. (Plan & Discussion)
[x] Docs docs docs docs....
[x] Adjust for middleware interface for 1.0
[x] Start working on ReplicationQueue

If none of the above tasks speak to you then please star this repo and hang tight until middleware docs have been written. If you already use multifeed then your application will hopefully be decentstack powered soon If you're using something else then please let me know and I'll submit PR's where it makes sense.

Any other questions or concerns, please submit a comment or issue, and

thank you for your time and energy! :heart:

ameba23 commented 4 years ago

Exciting stuff! Great to see this is developing. and also like that it is storage agnostic!

one thing i noticed recently with regards to wanting to have more control over who we replicate with - i noticed hypercore's replicate method already takes a keypair for the noise protocol as opts. https://github.com/mafintosh/hypercore#var-stream--feedreplicateisinitiator-options

perhaps noise protocol handshaking would give us the same kind of authentication for when replicating that we are hoping to get from what was replic8. im not saying this is a duplication of efforts - rather its something to play nicely with / use to our advantage. (maybe this is for another issue)

keep it up!

telamon commented 4 years ago

@ameba23 The keypair and authenticated replication is something I to am very excited about, and it should be exposed as the same opts that hypercore supports. (If hypercore still just forwards replicate(opts) options to the protocol#open() then it's a nobrainer to do the same thing with decentstack.)

Aside from exposing standards interfaces, I'm a bit curious if it would make sense to provide an higher level api for setting keypairs through the middleware interface.

Either way I would like to hear more opinons and thoughts on this but i haven't gotten around yet to creating a separate issue.

decentstack / decentstack