blind-oracle / psql-streamer

Stream database events from PostgreSQL to Kafka
Mozilla Public License 2.0
59 stars 13 forks source link

possible timeout occured on pgx connection #3

Open timsolov opened 3 years ago

timsolov commented 3 years ago

I have many errors in log like:

2021/09/27 00:47:25.613785 Source-PSQL (db1): Replication error (walPositionPersisted: 290579576): replication failed: nil message received, should not happen

And in postgresql logs:

021-09-26 22:01:08.808 UTC [1834] ERROR:  replication slot "db1" already exists
2021-09-26 22:01:08.814 UTC [1834] LOG:  starting logical decoding for slot "db1"
2021-09-26 22:01:08.814 UTC [1834] DETAIL:  Streaming transactions committing after 0/14AC5220, reading WAL from 0/14BA1F70.
2021-09-26 22:01:08.815 UTC [1834] LOG:  logical decoding found consistent point at 0/14BA1F70
2021-09-26 22:01:08.815 UTC [1834] DETAIL:  There are no running transactions.
2021-09-26 22:01:21.908 UTC [1844] ERROR:  replication slot "db1" already exists
2021-09-26 22:01:21.913 UTC [1844] LOG:  starting logical decoding for slot "db1"
2021-09-26 22:01:21.913 UTC [1844] DETAIL:  Streaming transactions committing after 0/14AC5220, reading WAL from 0/14BA1F70.
2021-09-26 22:01:21.914 UTC [1844] LOG:  logical decoding found consistent point at 0/14BA1F70
2021-09-26 22:01:21.914 UTC [1844] DETAIL:  There are no running transactions.

The investigation leads me to Start method of Subscription struct in pgoutput package: https://github.com/blind-oracle/pgoutput/blob/master/sub.go#L98 which calls from https://github.com/blind-oracle/psql-streamer/blob/master/source/postgres/postgres.go#L286

It looks like psql-streamer breaks a connection to database but I'm not sure. Or pgx does it. Have you seen an error similar to mine? How did you solve it?

blind-oracle commented 3 years ago

Hi, no, I haven't seen this behavior, it looks a bit like a network issue, have you tried running both DB and service on the same host?

And I haven't really checked compatibility with recent Postgres versions, last was 10.x I think.