search

mkabilov / pg2ch

Data streaming from postgresql to clickhouse via logical replication mechanism
MIT License
195 stars 35 forks source link

Replication Failed Timeout 10s #40

Open stvhanna opened 2 years ago

stvhanna commented 2 years ago

@mkabilov Hi Murat, thanks for creating a useful tool! It runs successfully (replicates the rows from PG to CH) but the timeouts (doesn't keep listening for new inserts) after 10s. See log below. Any ideas why this is happening or how can it be fixed?

I'm using PG v14.2 and CH v22.2.3.5. 

pg2ch $ go run main.go --config config.yaml
2022/02/28 17:57:12 consuming changes for table testtable starting from 0/1C7B250 lsn position
2022/02/28 17:57:12 generation_id: 5
2022/02/28 17:57:12 Starting from 0/1C7B250 lsn
2022/02/28 17:57:12 Primary Keepalive Message => ServerWALEnd: 0/1C7A520 ServerTime: 2022-02-28 17:57:12.383815 -0500 EST ReplyRequested: false
2022/02/28 17:57:12 XLogData => WALStart 0/1C7A670 ServerWALEnd 0/1C7A670 ServerTime: 2022-02-28 17:57:12.414696 -0500 EST WALData size 21
2022/02/28 17:57:12 XLogData => WALStart 0/0 ServerWALEnd 0/0 ServerTime: 2022-02-28 17:57:12.415047 -0500 EST WALData size 326
2022/02/28 17:57:12 XLogData => WALStart 0/1C7A670 ServerWALEnd 0/1C7A670 ServerTime: 2022-02-28 17:57:12.415365 -0500 EST WALData size 325
2022/02/28 17:57:12 XLogData => WALStart 0/1C7B280 ServerWALEnd 0/1C7B280 ServerTime: 2022-02-28 17:57:12.4154 -0500 EST WALData size 26
2022/02/28 17:57:12 Primary Keepalive Message => ServerWALEnd: 0/1C7B2B8 ServerTime: 2022-02-28 17:57:12.419801 -0500 EST ReplyRequested: false
2022/02/28 17:57:22 replication failed: receive message failed: read tcp [::1]:64963->[::1]:5432: i/o timeout

It looks like the error is generating on https://github.com/mkabilov/pg2ch/blob/fb8b6ff6628b7808f6c3e18b60f32375b67f0523/pkg/consumer/consumer.go#L147