search

instructure / jsoncdc

Takes a logical decoding stream from postgresql and outputs it in a pluggable and safe way
62 stars 4 forks source link

PostgreSQL 10/11/12/13 Support? #8

Open jmealo opened 4 years ago

jmealo commented 4 years ago

Hello,

Are there any plans to support newer versions of PostgreSQL?

Thanks, Jeff

jmealo commented 3 years ago

Is this project still being maintained?

Mythra commented 3 years ago

Hey @jmealo ,

I was one of the maintainers on this when I used to work at Instructure (about 16 months ago now), I know there was at least some initial maintenance effort being put forth after I left as I wanted to make sure of that since I had put a lot into it.

That being said, I think this project was only maintained for a couple months after I left. Based on the (lack of) response on this issue (and minor asking around!), I don't believe this is being used at Instructure anymore as the increase of usage in Amazon's RDS which offers a different logical decoder made the maintenance effort into this not worth it anymore for the team I handed off too.

I have been away from the PostgreSQL scene for a few years now, so I'm not sure I should be taking up lead maintainer again, but I'll try to see what can be done to salvage.

jmealo commented 3 years ago

@Mythra: thanks for the reply. I figured as much. I'm disappointed that wal2json became the defacto. I lobbied RDS to include jsoncdc it but it didn't go anywhere. I still assert that jsoncdc should be the most accurate. I had a few projects I wanted to open source that depended on jsoncdc and plv8 and was surprised how quickly things decayed.

I'm not sure how much effort it's worth, I think implementing accurate pgoutput parsers in more languages might be the best way to go for interoperability with multiple clouds. I may focus on that but jsoncdc does all the heavy lifting for you if your destination is JSON anyway.

The database should know best how to convert its internal types to JSON. That's why I think jsoncdc was the way to go.

I would like to compare any new solutions to jsoncdc and use it as the benchmark for correctness; if I need to build a special 9.6 vm and find a way to build it, it's not the end of the world.

Thanks again for your work and explanation!

Mythra commented 3 years ago

Yeah I am also disappointed this didn't quite hit the mark. I certainly liked the project, and know when I was at Instructure doing some lobbying for JSONCDC in RDS too, to no avail.

Future looking I think if you wanted to give it a go again using JSONCDC as a base for accuracy I think is a fair judgement call. Ideally if I were still a maintainer of this, and was tasked with redoing it I'd probably do the following:

I think the biggest reason in retrospect jsoncdc never really took off is cause it was less well known, never out of testing pgxn (no easy install like wal2json had), and was pretty much just there as the code was. So if you knew about it, and you knew why it was good you liked it more, but for most people they'd either:

  1. Never get to the codebase.
  2. Not understand what the trade-offs were once they got to it.
  3. Not find an easy integration path even if they understood the trade-offs.

And if it were to succeed I think it not only needs to solve those more social problems, but also add a little technical benefit (such as being able to choose encoding), in order to supplant the existing solutions out there.

jmealo commented 3 years ago

@Mythra if you decide to dust this off, this seems helpful: https://justatheory.com/2020/10/release-postgres-extensions-with-github-actions/

Mythra commented 3 years ago

I'd probably never just do it myself 😅 I'd probably want at least one other person contributing. But hopefully someone with more resources feels inspired 😄