Closed ericmigi closed 5 years ago
Oh or did something get changed in the bridge to extract more scrollback upon reconnect @tulir?
It already does deduplication (required for any multi-user bridge), whatsapp has decent message IDs so it uses those. This looks like a bug:
mautrix-whatsapp | [May 17, 2019 00:34:54] [Database/Message/WARN] Failed to insert 13177303794-1502469171@g.us@5E60B05E0039C82AA3C4: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:54] [Portal/13177303794-1502469171@g.us/DEBUG] Handled message 5E60B05E0039C82AA3C4 -> $15580532943046GnqtR:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 14157896163-1473014025@g.us@3AC4E90B84749A0E3F87: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/14157896163-1473014025@g.us/DEBUG] Handled message 3AC4E90B84749A0E3F87 -> $15580532943050UaiTn:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3EB0D31900440DACD029: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3EB0D31900440DACD029 -> $15580532943048jQVEe:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3EB02CC8656802BF2A1F: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3EB02CC8656802BF2A1F -> $15580532943049wzNXo:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3AD06B1A32579A81541B: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3AD06B1A32579A81541B -> $15580532943051DYpnP:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 13177303794-1502469171@g.us@5E6D40B1292990EF359B: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/13177303794-1502469171@g.us/DEBUG] Handled message 5E6D40B1292990EF359B -> $15580532943044wMWNx:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3A4C54E46BA82BF485F9: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3A4C54E46BA82BF485F9 -> $15580532943052wVZmN:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 13177303794-1502469171@g.us@5EC70B4A7FAE6E4C054C: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/13177303794-1502469171@g.us/DEBUG] Handled message 5EC70B4A7FAE6E4C054C -> $15580532943045ZIXSV:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3EB0AAD0A31A9780A774: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3EB0AAD0A31A9780A774 -> $15580532953055JhOCb:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3EB065CB1009D868AF98: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3EB065CB1009D868AF98 -> $15580532953058ctUpJ:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3EB0B23BA0611DD05908: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3EB0B23BA0611DD05908 -> $15580532953056hyGdJ:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:55] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3AAF08D7BB29401B791D: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:55] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3AAF08D7BB29401B791D -> $15580532953057tKRmH:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:56] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3A487602E91C655EA555: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:56] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3A487602E91C655EA555 -> $15580532953059mPGGb:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:56] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3A2C455AA3EBB77BF31D: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:56] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3A2C455AA3EBB77BF31D -> $15580532953060JvVzm:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:56] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3EB048F6CD93BBF33583: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:56] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3EB048F6CD93BBF33583 -> $15580532963061oaQAK:pulsar.im
mautrix-whatsapp | [May 17, 2019 00:34:56] [Database/Message/WARN] Failed to insert 14124781086-1541440519@g.us@3A850A812EAE5B6274D1: pq: duplicate key value violates unique constraint "message_pkey"
mautrix-whatsapp | [May 17, 2019 00:34:56] [Portal/14124781086-1541440519@g.us/DEBUG] Handled message 3A850A812EAE5B6274D1 -> $15580532963062VWGGM:pulsar.im
I might have forgot to check the db in the deduplication process
ok this just happened again, same situation
Looks like a bug somewhere in the DB code: pq: sorry, too many clients already
. Theoretically could be a too low postgres connection limit, but it's far more likely that the bridge is not releasing some connections that it should release.
Oh, actually it might just be that the connection count isn't limited by default, and it's overflowing when it receives over a hundred messages at once.
pretty annoying bug since I had to go through all the chats and mark them as read. I think it may be necessary to start thinking about making the bridge a bit smarter about which messages it sends/receives. Perhaps take a hash of each message with timestamp and check to see if it's already been bridged over to matrix?
all these rooms should have been zero unread upon reconnecting