search

mannemsolutions / pgarrow

Stream Postgres through Kafka or RabbitMQ
GNU General Public License v3.0
2 stars 0 forks source link

Start enqueue with skip #35

Open RobertBlok opened 1 year ago

RobertBlok commented 1 year ago

When a transaction must be skipped from the CDC process, the enqueue process should have the possibility to start and skip a transaction.

pgarrow-pgarrowkafka-1 | 2022-12-28T09:20:18Z fatal pgarrow does not work (yet) with values like map[customer:Alex Cross items:map[product:Tea qty:6]].(map[string]interface {}) pgarrow-pgarrowkafka-1 exited with code 1

sebasmannem commented 1 year ago

This specific thing (special types like JSON) is something we should try to fix entirely. And I added an option to skip specific postgres errors (example is SQLSTATE 23505: duplicate key value violates unique constraint). If we really wanna have an option 'to start over', we should also have an option to refresh the data, since 'just skip messages on the queue' would be the same as 'lets introduce dataloss'. Keeping this ticket open for nu, until we have a 'start over properly' option and then we can close this one too...

sebasmannem commented 1 year ago

'start over properly' option would be pat of #4

sebasmannem commented 1 year ago

JSON now supported.

4 is now being worked on.

Please test with next release...