search

duckdb / duckdb_mysql

https://duckdb.org/docs/extensions/mysql
MIT License
55 stars 13 forks source link

High Memory Usage During MySQL Table Import (DuckDB 0.10.0, Ubuntu) #59

Open jaldaadithya opened 7 months ago

jaldaadithya commented 7 months ago

What happens?

When I try to create a table in the duckdb from the mysql table which was around 15 GB, the duckdb process is taking entire system memory (around 30 GB) and the system has 32GB RAM.

To Reproduce

I attached mysql using mysql_scanner 1) ATTACH 'host=host user=user port=3306 database=duckdb password=password socket=/var/run/mysqld/mysqld.sock' AS mysql_db (TYPE mysql_scanner,READ_ONLY); 2) SET threads = 4; SET memory_limit = '4GB'; SET max_memory = '4GB'; SET mysql_tinyint1_as_boolean=false; SET old_implicit_casting = true; 3) CREATE OR REPLACE TABLE table AS FROM mysql_db.table; the above table was around 15GB. 4) I tried using LD_PRELOAD too.

OS:

Ubuntu

MySQL Version:

8.0.36

DuckDB Version:

0.10.0

DuckDB Client:

jdbc

Full Name:

Adithya

Affiliation:

OvalEdge

Have you tried this on the latest main branch?

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

yinheli commented 7 months ago

Same issue here, How to reduce the memory usage while import? @Mytherin

ulissescappato commented 5 months ago

Same issue here...

Thanks,

mendelfriedman commented 1 week ago

I'm seeing the same issue using mysql_query() and copy to file.

Any solutions here?