search

duckdb / duckdb-rs

Ergonomic bindings to duckdb for Rust
MIT License
508 stars 113 forks source link

Duckdb updates #337

Closed Swoorup closed 1 month ago

Swoorup commented 5 months ago
  1. Strong types for duck db vectors.
  2. Ensure correct size are reserved before writing to duckdb vectors.
  3. Introduce DataChunkHandle for manipulating data chunks, and wrap unsafe functions at bay. There are still few handle like duckdb_appender that is better off being wrap instead of sprinkling unsafe everywhere.
  4. Add union, struct type and nested type without the duplicated code. (It should be simple enough to now add other nested types like Map)
  5. Appender keeps a cached copy of the schema for error handling and insertions.
  6. Fix large amount of record insertion (Respect the duckdb limit)
  7. Remove unsafe code for traits that need not be unsafe. Entire idea for rust is to limit usage of unsafe code when providing as a library.
  8. For unit testing, I also added arrow-convert library which makes it easier to derive arrow arrays directly from rust specific unions/structs than hand writing it.
Swoorup commented 5 months ago

Fixes or partially fixes the following:

era127 commented 5 months ago

I think the 306 update to the test case could be merged into your changes.

Swoorup commented 5 months ago

Union feature is blocked by a suitable release i.e which includes the fix https://github.com/duckdb/duckdb/pull/11326 https://github.com/duckdb/duckdb-rs/pull/336

Mytherin commented 4 months ago

Thanks for the PR! Could you perhaps split this up into smaller chunks so that it is easier to review?

Swoorup commented 1 month ago

NOTICE: I have abandoned this work. Anyone is welcome to take over and integrate bits and pieces they may find useful.