search

apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
https://arrow.apache.org/
Apache License 2.0
14.57k stars 3.54k forks source link

[C++] Native result set adapter for PostgreSQL / libpq #16693

Open asfimport opened 7 years ago

asfimport commented 7 years ago

We can look at https://github.com/MagicStack/asyncpg for inspiration

Reporter: Wes McKinney / @wesm

Note: This issue was originally created as ARROW-1106. Please see the migration documentation for further details.

asfimport commented 5 years ago

Neville Dipale / @nevi-me: I recently saw https://github.com/heterodb/pg2arrow, which might be interesting to look at

asfimport commented 5 years ago

KaiGai Kohei: What is the expected timeline for this project?

I'm interested in contributing the pg2arrow code for Apache Arrow project.

Probably, we can separate the above code into two portions; routines to read/write files in Arrow format and to interact PostgreSQL using libpq. The former portion has been also used for PostgreSQL connector (Arrow_Fdw), so it is preferable to package as an independent library for native-C language.

On the other hands, I have to focus on software quality improvement of PG-Strom for the next 2-3 months. I expect the progress will not be quick enough.

 

asfimport commented 5 years ago

Wes McKinney / @wesm: It is not a short term priority for anyone as far as I am aware. We can of course discuss further on the mailing list

My particular interest would be making this software accessible to users of the Arrow libraries in languages like Python and R