search

instructure / jsoncdc

Takes a logical decoding stream from postgresql and outputs it in a pluggable and safe way
62 stars 4 forks source link

How does jsoncdc handle large transactions? #6

Closed taybin closed 5 years ago

taybin commented 5 years ago

Reading through wal2json's issues, I see they have an option to write the json in chunks, so that large updates don't reach the default 1GB memory limit in postgresql.

https://github.com/eulerto/wal2json/issues/100#issuecomment-459399625

How does jsoncdc deal with this?

Mythra commented 5 years ago

Hey TayBin,

Sorry about the delay getting you an answer here. To be honest jsoncdc doesn't do anything to handle this yet (and so you will hit the memory limit). We've been trying to decide internally what the best option to take here is.

We have some concerns about doing the chunks approach that wal2json has in our architecture, but unfortunately we don't have any concrete plans yet on how exactly we want to handle this.

Sorry I couldn't give some better news!

taybin commented 5 years ago

FWIW, wal2json is shipping a new json format soon that should address some of the concerns. No idea what it looks like yet, but it might be something to look at when it ships. Thank you!