search

duckdb / pg_duckdb

DuckDB-powered Postgres for high performance apps & analytics.
MIT License
1.65k stars 60 forks source link

Does this extension support concurrent access to duckdb from multiple connections? #134

Closed chrisfw closed 3 months ago

chrisfw commented 3 months ago

Hello,

I am very excited by the prospect of using this extension and combining two amazing projects!

That said, during experimentation with duckdb_fdw, I discovered that using multiple concurrent connections from separate processes was not supported. This seemed like an inherent problem with using an embedded db in a multi-process environment, so I am wondering if this will somehow be addressed architecturally by this extension implementation?

Thanks, Chris Whelan

JelteF commented 3 months ago

Yes, supporting multiple concurrent connections is definitely something we support.

Because we currently don't support store any data in DuckDB, but only use its execution engine and read_csv/read_parquet functions. We simply create one DuckDB instance per connection.

This approach is something that we want to change though to make resource management easier, and because we want to support storing data in DuckDB itself. We plan on using a Postgres BackgroundWorker for this. Then we'll load DuckDB there and communicate with this process from the client processes using IPC (sockets and/or shared memory).

chrisfw commented 3 months ago

@JelteF , thank you for providing this info, that is great news! I am very excited to test this extension out and also to hear that in the future the integration will support storing data in duckdb as well!

Regards, Chris