search

duckdb / duckdb_mysql

https://duckdb.org/docs/extensions/mysql
MIT License
55 stars 13 forks source link

COPY from Parquet seems to keep the Parquet file open (so the file cannot be deleted) #100

Open pronzato opened 1 week ago

pronzato commented 1 week ago

What happens?

After copying a Parquet file into a MySQL table, the Parquet file seems to be locked and therefore cannot be deleted until the process is killed. We use tmp Parquet files to bridge data from 1 system into MySQL so once we've ingested the Parquet file into MySQL we need to delete it.

To Reproduce

CALL mysql_execute('mysqldb', 'CREATE TABLE t1 (i INTEGER);'); COPY mysqldb.t1 FROM '/tmp/t1/data.parquet'

File f = new File("/tmp/t1/data.parquet"); boolean deleted = f.delete(); System.out.println("deleted = " + deleted);

OS:

Windows

MySQL Version:

8.1

DuckDB Version:

1.1.1

DuckDB Client:

Java

Full Name:

Gianni Pronzato

Affiliation:

pronzato

Have you tried this on the latest main branch?

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

pronzato commented 1 week ago

Same happens if you try: INSERT INTO mysqldb.t1 FROM read_parquet('/tmp/t1/data.parquet');