search

pacman82 / arrow-odbc

Fill Apache Arrow record batches from an ODBC data source in Rust.
MIT License
52 stars 10 forks source link

Polars and arrow-odbc #95

Closed lpj145 closed 7 months ago

lpj145 commented 7 months ago

Hey @pacman82 thanks for your job you have done a incredible work here, please did you have some articles or examples of how to integrate this crate with polars? I had find some posts talking about this crate with polars but no single one example how to do it.

See ya!

pacman82 commented 7 months ago

Hello @lpj145 , thank you for the kind words. As far as I know the Python binding of Polars are integrating with the Python bindings of this crate. See: https://docs.pola.rs/py-polars/html/reference/api/polars.read_database.html. These bindings are maintained by the Polars developers.

The Polars Rust crate has no direct support. People have converted it in the past still: https://github.com/pacman82/odbc-api/discussions/536#discussioncomment-8807348

Also arrow2 has direct ODBC bindings, which like arrow-odbc are build on top of odbc-api and use bulk fetches. However I would not recommend using them. They are more a result of a one day hack and unmaintained.

If this occurs more often I could also expose the C-Interface in the Rust crate, to save one marshaling step. Which arrow implementation does Polars use these days? Is it arrow2 or something else?

Best, Markus

lpj145 commented 7 months ago

Polars uses https://crates.io/crates/polars-arrow I think it is his own implementation, by the way it's is a fork from arrow2.

pacman82 commented 7 months ago

Given that arrows main selling point is that it can share binary representations the community is weirdly insistent of marshaling the meta-information into all kinds of formats 😅 . The C-Representation is good enough. I wish we could just use that as the canonical representation and just wrap it into some binary identical safe types.

ugoa commented 7 months ago

Hey Markus @pacman82 , big fan of your work here. Just want to say hi 🙂

I once made a attempt to update the odbc-api version used by arrow2, but only realized later that the project itself went archived and merged into the official arrow project 😂. Nevertheless I learned a lot by studying your code. Thanks a lot for all the work you have done!

pacman82 commented 7 months ago

Hello @ugoa , thank you, you are welcome! I learn a lot myself through these projects. I did not realize myself until now arrow2 and the official arrow merged. Yet I think it is a good thing to not split the community to granular.