Currently, the Appender operates row-at-a-time with duckdb_append_TYPE. However, DuckDB performs significantly better with bulk appends, i.e., with appending whole data chunks. The C API offers duckdb_append_data_chunk.
Additionally, there is no support for nested types in the Appender, and while duckdb supports appending Value in the C++ API (DUCKDB_API void BaseAppender::Append(Value value)), it does not offer the same in the C API.
We want to pick this up and rework the Appender to use duckdb_append_data_chunk and eventually allow appending nested types by building these data chunks recursively. I've decided to open this issue to make potential other developers aware that there is work on this. I hope that's fine. :) I will close it once there is an initial PR.
marcboeker
commented
8 months ago
Currently, the
Appenderoperates row-at-a-time withduckdb_append_TYPE. However, DuckDB performs significantly better with bulk appends, i.e., with appending whole data chunks. The C API offersduckdb_append_data_chunk.Additionally, there is no support for nested types in the
Appender, and while duckdb supports appendingValuein the C++ API (DUCKDB_API void BaseAppender::Append(Value value)), it does not offer the same in the C API.We want to pick this up and rework the
Appenderto useduckdb_append_data_chunkand eventually allow appending nested types by building these data chunks recursively. I've decided to open this issue to make potential other developers aware that there is work on this. I hope that's fine. :) I will close it once there is an initial PR.Affiliation: DuckDB Labs