apache / arrow-flight-sql-postgresql

Apache Arrow Flight SQL adapter for PostgreSQL
https://arrow.apache.org/flight-sql-postgresql/
Apache License 2.0
64 stars 9 forks source link

Fix bulk ingestion benchmark #171

Closed kou closed 7 months ago

kou commented 7 months ago

In the current implementation, PostgreSQL based approaches don't include time to construct query (INSERT ... VALUES (...) ... (...) and COPY data). But we should include time for it. It's needed time for bulk ingestion because the target data in memory are not represented as texts for INSERT .... For example, numbers are represented as int32_t and so on. Bulk ingestion needs to convert in memory data to query.