Roadmap - Githubissues

jemc / jylis

A distributed in-memory database for Conflict-free Replicated Data Types (CRDTs). :seedling: :left_right_arrow:

https://jemc.github.io/jylis

Mozilla Public License 2.0

75 stars 6 forks source link

Roadmap #1

Open jemc opened 6 years ago

jemc commented 6 years ago

1.0 Milestone

[x] Basic data types
[x] Client access
[x] Cluster membership add/remove
[x] Data replication on update
[x] Distributed cluster log
[x] Version-independent cluster serialization
[x] Disk persistence
[x] Shared causal contexts
[x] Anti-entropy repair on node join
[ ] Continuous anti-entropy repair
[ ] Finalize cluster setup operational controls
[ ] Compaction of removed cluster members' data
[ ] Cluster and disk protocol compatibilty plan.
[ ] Lock Pony language and library revisions for reproducible builds.
[ ] Clear out "TODO" markers in code and documentation; replace all doc stubs.

Performance

[ ] Profile, benchmark and optimize cluster serialization; consider less human-readable protocols.

Beyond

[ ] Removing entries from keyspaces
[ ] User-creatable keyspaces?
[ ] Causal transactions
[ ] Client subscriptions
[ ] Keyspace iteration and search
[ ] Keyspace filtering

creatorrr commented 5 years ago

@jemc jylis is a really neat implementation. What's the current status of the project? Is the library still under active development?

jemc commented 5 years ago

At this time, the work is on pause. I am looking for an opportunity to be sponsored to work on this as a paid project before I invest more time into it.

creatorrr commented 5 years ago

Gotcha. Thanks for the update @jemc. We might be interested in sponsoring the project down the road but our budget for this quarter has already been decided. I'll keep you posted on this. At the moment, would you consider the library to be in a state that we could play with it and safely deploy it in production?

jemc commented 5 years ago

It depends on your use case. There are two potential missing roadmap pieces to be aware of:

I haven't done extensive performance optimizations, so while it's pretty fast as-is, there may be some unconsidered cases that are slow.
The disk storage protocol is still going to be subject to change before 1.0. It's not terribly optimized at this point and not even guaranteed to be stable. So I wouldn't recommend using pre-1.0 jylis for use cases where data loss is unacceptable - for now, I'd prefer it for in-memory ephemeral-data use cases where data loss from time to time may be okay.

That said, if you decide to move forward, I'd love to get feedback from you about how you're using the product and which parts of it are compelling (and which parts may be annoying or troublesome).

creatorrr commented 5 years ago

Gotcha.

I'd love to get feedback from you about how you're using the product and which parts of it are compelling

Absolutely! I would be happy to share more once we have a concrete implementation plan but basically, we are building a horizontally-scalable component that requires extremely high fault-tolerance. We are using ponylang to write it in and looking for a replicated ephemeral storage solution. We are looking at Apache Ignite and now jylis as well.

I will keep you posted about both our experience with jylis and sponsoring the project (that will take some time, however). In the meantime, please feel free to drop me a line directly at singh@diwank.name if you'd like to learn more about our use case.