duckdb / duckdb_mysql

https://duckdb.org/docs/extensions/mysql
MIT License
55 stars 13 forks source link

Duckdb mysql extension took extreme longer time to access data than other components #95

Open dawsondong opened 3 weeks ago

dawsondong commented 3 weeks ago

Based on such TPCH test background: Table: orders Data Amount: 150,000,000 SQL to use duckdb mysql extension: copy my.orders to '/tmp/orders.mysql.duckdb.parquet' (format parquet); Time to consume: 534 seconds.

SQL to use Doris: select * from my.test.orders into outfile "file:///tmp/orders.mysql.parquet" format as parquet; Time to consume: 122 seconds.

With simple JDBC application to access orders table and write, it will consume about 150 seconds which is closed to Doris.

So in most of cases, duckdb_mysql takes over 4 times cost to achieve same task. It could get improved.