search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.15k stars 842 forks source link

Add JetStream support #646

Closed smoothd closed 3 years ago

smoothd commented 3 years ago

JetStream will be NATS Streaming's successor, replacing it eventually. JetStream is supported by jsm.go.

It would be great to have jsm Inputs and Outputs analogous to today's nats_streaming support.

[Edit] I'm happy to collaborate on this and to provide an initial PR if wanted.

Jeffail commented 3 years ago

Hey @smoothd, sounds good. The NATS APIs are usually very clean so I'm happy to add experimental support even if they're subject to change.

bkcsfi commented 3 years ago

@smoothd are you still interested in Jetstream for Benthos?

It looks like the go client now has some jetstream support https://github.com/nats-io/nats.go but the api is a bit different than jsm.go

There are still some issues to be fixed..

Jeffail commented 3 years ago

It's happening https://github.com/Jeffail/benthos/tree/nats-jetstream, still waiting for a tagged release of https://github.com/nats-io/nats.go containing the new APIs before I merge it but I'll have the connectors prepared.

bkcsfi commented 3 years ago

awesome, thank you very much for working on this!

Jeffail commented 3 years ago

Awesome, they tagged a release https://github.com/nats-io/nats.go/releases/tag/v1.11.0, and now it's in: https://github.com/Jeffail/benthos/commit/90b2a62dfb204fd621a4392b22d4b438fcea928f

I'll put a release out soon.

smoothd commented 3 years ago

@Jeffail Wonderful, thank's a lot for the effort. Will take some time to try it out though since we still run on nats-streaming.