feat: introduce pgoutput plugin

replicase / pgcapture

A scalable Netflix DBLog implementation for PostgreSQL

Apache License 2.0

220 stars 31 forks source link

feat: introduce pgoutput plugin #30

Closed KennyChenFight closed 1 year ago

KennyChenFight commented 1 year ago

This is a draft version for support pgoutput plugin when pg version >= 14.

implement pgoutput plugin decoder (only support binary format)
- ref message format: https://github.com/postgres/postgres/blob/master/src/backend/replication/logical/proto.c
- ref pgoutput decode: https://github.com/postgres/postgres/blob/master/src/backend/replication/pgoutput/pgoutput.c
- separate pglogical and pgoutput decoder implementation
setup ci for pg14
- setup pg14 logical docker image and use pglogical patch to remove filter and local node
- use PG_VERSION env to decide should run pgoutput test or not

rueian commented 1 year ago

This is really cool! Thanks @KennyChenFight!

BTW, is there any chance to drop the pglogical dependency?

KennyChenFight commented 1 year ago

This is really cool! Thanks @KennyChenFight!

BTW, is there any chance to drop the pglogical dependency?

Now, the client has the option to choose which decoder to use. By default, the pglogical decoder is used. However, if the client is using PostgreSQL version 14 or above, they can specify to use the pgoutput decoder instead. In this case, the client does not need to install pglogical in their PostgreSQL. So, we still can keep pglogical decoder implemation in code ?

rueian commented 1 year ago

Sure, we should keep the pglogical implementation. However, I believe making pgoutput as default is better since it lowers the barrier to use this library for new users.

KennyChenFight commented 1 year ago

Sure, we should keep the pglogical implementation. However, I believe making pgoutput as default is better since it lowers the barrier to use this library for new users.

Considering that the pgoutput binary format is only supported in PG 14 or above, and PG 14 is still a relatively newer version, it might be better to set the pglogical decoder as the default option. Or, maybe I could implement pgoutput text format ?

rueian commented 1 year ago

It is ok to only support PG 14 and above. I believe there are more people using PG >= 14 than having pglogical installed.

KennyChenFight commented 1 year ago

@rueian I think you can start to review my code !