search

delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake
Apache License 2.0
337 stars 72 forks source link

Possibility of adding Python bindings to this project #124

Closed MrPowers closed 1 year ago

MrPowers commented 1 year ago

Would it be possible to add Python bindings to this project?

I think a lot of programmers are comfortable writing Python, but aren't comfortable with Rust. I'm assuming the Python bindings could be added like they were for the delta-rs project. Thoughts?

houqp commented 1 year ago

It definitely is possible and should be pretty easy to implement using pyo3. However, given that the project is supposed to be consumed as a binary, it might not provide much value in doing so. If you have a particular use-case in mind, it would be very useful to share it here.

rtyler commented 1 year ago

I'm not sure what the point of that would be @MrPowers ? What's the use-case?

MrPowers commented 1 year ago

@rtyler - The high-level idea here is how can we make this project super easy to install and how can we can attract more users.

Right now, the setup instructions are a little scary. It'd be great if we could make them easier and then start marketing this project to a wider audience.

The mdBook setup instructions were "download the pre-compiled binary & add it to your path". I think that's comparatively less scary, but still a little technical. Lots of users don't know how to add binaries to their PATH.

Perhaps we could offer users a brew install delta-kafka-ingest installation experience?

My suggestions might be making no sense here, haha. But would be cool if there was a real simple story I could tell about why people should definitely be using this lib!

rtyler commented 1 year ago

@MrPowers I understand what you want, but Python bindings won't do that. I'm going to close this and create a new issue to provide simple install and getting started instructions.