search

marcboeker / go-duckdb

go-duckdb provides a database/sql driver for the DuckDB database engine.
MIT License
623 stars 97 forks source link

Add support for `duckdb_arrow_scan` and `duckdb_arrow_array_scan` #161

Open disq opened 6 months ago

disq commented 6 months ago

Extend the Apache Arrow interface support to expose duckdb_arrow_scan and duckdb_arrow_array_scan functions. (seems like only duckdb_query_arrow_schema is supported for now)

duckdb_arrow_scan duckdb_arrow_array_scan

This would be useful for fast data loading directly from Arrow types.

marcboeker commented 6 months ago

Hi @levakin, what would be your opinion on this? Is it something we can easily add?

levakin commented 6 months ago

hi @marcboeker , it should be doable. The only tricky thing is Note that after running duckdb_arrow_array_scan, duckdb_destroy_arrow_stream must be called on the out stream. this part. Maybe it can implemented using a pattern with something similar to cancel function being returned.

marcboeker commented 6 months ago

Thanks @levakin for the evaluation. Would it make sense to add a method that returns an ArrayScanner which has to be closed afterwards?

disq commented 5 months ago

I wanted to take a look at this the other day, but didn't know where to start.