search

liftbridge-io / liftbridge

Lightweight, fault-tolerant message streams.
https://liftbridge.io
Apache License 2.0
2.57k stars 107 forks source link

Contributing #53

Closed jmwilkinson closed 4 years ago

jmwilkinson commented 5 years ago

Hi, how open is this project to outside contributions?

Related, is there an expectation it will ever be merged back into "mainline" nats work?

Is development here ongoing? Does it relate in any way to the mention of "Jetstream" on the nats-io project page?

tylertreat commented 5 years ago

Hi, outside contribution is definitely welcome! If it's mostly minor changes, feel free to make PRs directly. If you're interested in working on larger changes or major features, all I ask is you open an issue first describing your plan.

There is no expectation Liftbridge will ever be merged into mainline NATS work. Liftbridge is a standalone project.

Development is indeed ongoing. I'm currently working on key compaction.

Regarding Jetstream, a little bit of history which will hopefully help clarify things. "Jetstream" was actually my original name for this project as I was conceptualizing it while I was a member of the NATS team. I have since moved on from the NATS team and I decided to rename this project to Liftbridge to avoid confusion with whatever the NATS team has planned for Jetstream. As such, Liftbridge and Jetstream are completely unrelated projects and are not affiliated with each other. I can't speak to Synadia's plans for Jetstream. I can, however, speak to the plans for Liftbridge. :)

jmwilkinson commented 5 years ago

Ok, thanks.

I really like the technical direction of this project. I read your blog posts on Jetstream about a year ago and they really resonated with my own opinions what a performant distributed commit log should look like, after having used both Kafka and nats.

Time permitting, I would like to contribute the the embedded nats option... I think simplicity of setup should be a core objective, and while a nats cluster is relatively easy to start up, it's still another thing to manage.

To that end, I've investigated what it would take to actually embed the "nats" portion of the server, and the difficult thing seems to be how coupled liftbridge currently is with the nats connection. My initial approach was to try and abstract the Conn behind an interface, but I ran into problems with the nats transport which explicitly requires a nats conn. Of course, anything can be refactored an interfaces are easily introduced in Go, but I was wondering if you have another approach in mind?

tylertreat commented 5 years ago

My initial plan was to model it after what NATS Streaming does since NATS Streaming also supports an embedded mode of NATS. This allows it to just use Conn as usual and seems like the simplest approach. https://github.com/nats-io/nats-streaming-server/blob/247d7fa2c948ab918d7b4a9f761010082e8b92ea/server/server.go#L2083-L2107

The biggest pain/point of awkwardness seems like it's how best to deal with config for the embedded gnatsd.

jmwilkinson commented 5 years ago

It's funny, just to get something working my initial attempt did exactly that- just made the Liftbridge main start the nats server.

I think there are two problems with that approach, though.

  1. Cohesion. This ties in closely with dealing with the embedded config, but more broadly relates to needing to learn at least two different approaches, one for the producer and one for the consumer.
  2. Socket communication. It seems a bit silly to have the same process communicate over a socket rather than just internal to the process.

I'm kinda struggling with how much value nats as a whole brings, beyond the core of subject routing a management...

tylertreat commented 5 years ago

Yeah, it's a little clumsy I agree, though my primary goal for the project was to provide Kafka-like semantics for NATS, not to necessarily be a pure standalone Kafka replacement. But my vision for it was to allow it to run in a standalone fashion with NATS powering it but ultimately abstracted away, then provide a gRPC publish API. This way, legacy NATS clients could continue to use it but newer clients could switch over to the gRPC-based protocol.

tylertreat commented 4 years ago

Closing this issue since the embedded NATS work will be tracked in https://github.com/liftbridge-io/liftbridge/issues/19.

somombo commented 4 years ago

Regarding Jetstream, a little bit of history which will hopefully help clarify things. "Jetstream" was actually my original name for this project as I was conceptualizing it while I was a member of the NATS team. I have since moved on from the NATS team and I decided to rename this project to Liftbridge to avoid confusion with whatever the NATS team has planned for Jetstream. As such, Liftbridge and Jetstream are completely unrelated projects and are not affiliated with each other. I can't speak to Synadia's plans for Jetstream. I can, however, speak to the plans for Liftbridge. :)

FYI all the info about Synadia's plans for the official NATS Jetstream project can be found here https://github.com/nats-io/jetstream