search

citusdata / citus

Distributed PostgreSQL as an extension
https://www.citusdata.com
GNU Affero General Public License v3.0
10.15k stars 651 forks source link

Refactor out connection pooling parts of adaptive executor #3496

Open pykello opened 4 years ago

pykello commented 4 years ago

The usecase is that then we can use it for COPY too (if configured), so we can solve issues like https://github.com/citusdata/citus/issues/710 and https://github.com/citusdata/citus/issues/1047.

I am not sure how would it look like, if the suggestion makes sense, @serprex or I can start on creating an API proposal for this.

@marcocitus any reservations on this?

marcocitus commented 4 years ago

How is it different from the work you did to multiplex multiple COPY commands over the same connection?

pykello commented 4 years ago

@marcocitus In COPY multiplex, we don't assign connections, we just ask for already assigned connections. So it requires another DDL/DML being have run before the COPY to reduce number of connections used.

If COPY has to assign connections because it is the first command in the xact, it does the simplest thing and assigns one connection per shard.

I wanted the connection assignment (and maybe establishment) part to be refactored so COPY can also use it maybe, so modifications to our strategies in that regard is not duplicated in 2 code paths.

marcocitus commented 4 years ago

I think it would probably make sense for COPY to have its own specialized executor given how different COPY commands are from regular queries.