search

duckdb / duckdb-java

DuckDB JDBC Driver
https://duckdb.org/docs/api/java.html
MIT License
34 stars 21 forks source link

Missing jemalloc dependency #63

Open akoshchiy opened 2 months ago

akoshchiy commented 2 months ago

Seems that duckdb sources is not bundled with jemalloc. https://github.com/duckdb/duckdb/discussions/13075 It could be the reason of some memory issues, e.g. https://github.com/duckdb/duckdb/issues/9712

brianwyka commented 1 month ago

@lnkuiper , do you think this could be the issue?

lnkuiper commented 1 month ago

@akoshchiy, indeed, the JDBC client has not been bundled with jemalloc since we moved it out of the main repository to this repository. I don't see why it can't be bundled with jemalloc, so we should look into this.

@brianwyka, I'm not sure. I have been able to reproduce a similar memory retention issue, but only on MacOS regardless of whether jemalloc was bundled. On Linux, I was not able to reproduce this issue.

brianwyka commented 1 month ago

@lnkuiper , do you have instructions on how to bundle it? Perhaps I can contribute if its not on the roadmap or radar.

lnkuiper commented 1 month ago

You can enable jemalloc by adding it to the list of extensions in vendor.py and calling python3 scripts/vendor.py --duckdb=/path/to/duckdb/repo, and then make release in duckdb-jdbc repo.

For other clients, we only bundle jemalloc on x86 linux, and we'd like to keep it the same for JDBC, so it needs to check this. You can check how we do this in duckdb repo at tools/pythonpkg/setup.py.