search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.1k stars 1.36k forks source link

Databricks Delta Connector Request #4434

Open sridharpaladugu opened 10 months ago

sridharpaladugu commented 10 months ago

What motivated this proposal?

Hi. Lakehouse platforms are on the rise and became the defacto Data lakes. Delta Lake platform is much widely used and most popular in the segment. Also Delta is the default standard today for several Lakehouse implementations. So i am requesting for a native connector with Delta Lake (https://github.com/delta-io/delta). This help the large developer base of Delta to leverage NATS features easily.

What is the proposed change?

New Feature to integrate NATS.io with Delta lake (https://github.com/delta-io/delta)

Who benefits from this change?

This help integrate Delta lake and NATS developers to do pub-sub data operations with Delta lake.

What alternatives have you evaluated?

Looked at existing spark connector https://github.com/nats-io/nats-spark-connector

rtyler commented 10 months ago

:wave: @ripienaar , I work on the delta-rs (Rust and Python) bindings for Delta, I'm not sure whether this request requires a contribution on the client side (polyglot) or something in the nats core (golang). If the latter, I would be happy to make some time to chat about what we're doing in the Delta community to bring support to golang

ripienaar commented 10 months ago

Thanks for the offer @rtyler :)

We do not work on integrations like this as a general thing that we just do, this tend to happen as contributions or as comissioned work.

So while I think yeah this does look interesting, I am not sure what shape this would take exactly should such a thing exist but seems the server would not be the place for this, it would be a kind of connector similar to the spark one

jnmoyne commented 10 months ago

Thanks for the offer indeed! I would also say there shouldn't be to change anything on the server to create a binding, you would simply just use the nats client library to store and retrieve messages from Streams (for example (and not limited to) the Java library which you can also use from Scala).