psycopg / psycopg2

PostgreSQL database adapter for the Python programming language
https://www.psycopg.org/
Other
3.37k stars 505 forks source link

send_feedback logic question? #749

Closed jiamo closed 6 years ago

jiamo commented 6 years ago

Hi I am using seed_feedback in my python example to read logic replication data. There is problem after the seed_feedback was called. select * from pg_replication_slots; only show that confirmed_flush_lsn was changed. While the restart_lsn don't change. This may cause my postgres disk usage growing.

I find a debug error in log:

2018-07-23 17:19:35.869 CST [2255] DEBUG: failed to increase restart lsn: proposed 2F/56934F30, after 2F/56934F30, current candidate 2F/56934E50, current after 2F/56934E50, flushed up to 2F/56913378

and the slot stat is

postgres=# select * from pg_replication_slots;
LOG:  duration: 0.660 ms  statement: select * from pg_replication_slots;
 slot_name |  plugin  | slot_type | datoid | database | temporary | active | active_pid | xmin | 
 catalog_xmin | restart_lsn | confirmed_flush_lsn
 -----------+----------+-----------+--------+----------+-----------+--------+------------+------+--------------+-------------+---------------------
 wal2json  | wal2json | logical   |  12558 | postgres | f         | t      |       2255 |      |       953428 | 2F/563483E0 | 2F/56913378

My code pkt = self.cur.read_message() always got no pkt (There is no message anymore) And the last pkt.data_start is 203315819384 using

def int_lsn_to_str(lsn):
    return "%X/%08X" % ((lsn >> 32) & 0xFFFFFFFF, lsn & 0xFFFFFFFF)

is '2F/56913378'

So the problem seems to be I can't got data_start lsn.
And If there is no data change, the wal file is keep going to exist. I am using https://github.com/eulerto/wal2json.

jiamo commented 6 years ago

Hi. I find there is option include-lsn in wal2json that can fix my problem. https://github.com/jiamo/python-psql-replication/commit/cbed8e0edb2da8ea83fccb678e691eacc62a602e