Open melroy89 opened 1 year ago
do you get any more output showing what concurrent update is causing it? From memory I think there are usually a few more lines after that error message.
(Your issue is different to #3234.)
From the Postgres side, it logs like this:
2023-07-18 15:55:17.396 CDT [6670] ERROR: could not serialize access due to concurrent update
2023-07-18 15:55:17.396 CDT [6670] STATEMENT: INSERT INTO receipts_graph (room_id, receipt_type, user_id, thread_id, event_ids, data) VALUES ('!<room_id_hash_thingy>:matrix.org', 'm.read', '<random remote user_id', 'main', '["$N5W<...event_id>?"]', '{"thread_id":"main","ts":1689712423103}') ON CONFLICT (room_id, receipt_type, user_id, thread_id) DO UPDATE SET event_ids=EXCLUDED.event_ids, data=EXCLUDED.data
do you get any more output showing what concurrent update is causing it? From memory I think there are usually a few more lines after that error message.
(Your issue is different to #3234.)
No there are no other messages behind this error message..
2023-07-16 15:02:50,037 - synapse.federation.federation_server - 1470 - ERROR - PUT-181101 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
await handler(origin, content)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
await self._handle_new_receipts(receipts)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
res = await self.store.insert_receipt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
self.db_pool.simple_upsert_txn(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
2023-07-16 17:47:00,166 - synapse.federation.federation_server - 1470 - ERROR - PUT-190459 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
await handler(origin, content)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
await self._handle_new_receipts(receipts)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
res = await self.store.insert_receipt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
self.db_pool.simple_upsert_txn(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
2023-07-16 17:47:00,170 - synapse.federation.federation_server - 1470 - ERROR - PUT-190459 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
await handler(origin, content)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
await self._handle_new_receipts(receipts)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
res = await self.store.insert_receipt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
self.db_pool.simple_upsert_txn(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
2023-07-16 20:52:13,713 - synapse.federation.federation_server - 1470 - ERROR - PUT-206009 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
await handler(origin, content)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
await self._handle_new_receipts(receipts)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
res = await self.store.insert_receipt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
self.db_pool.simple_upsert_txn(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
2023-07-18 14:52:54,153 - synapse.federation.federation_server - 1470 - ERROR - PUT-331290 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
await handler(origin, content)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
await self._handle_new_receipts(receipts)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
res = await self.store.insert_receipt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
self.db_pool.simple_upsert_txn(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
2023-07-18 14:54:23,066 - synapse.federation.federation_server - 1470 - ERROR - PUT-331376 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
await handler(origin, content)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
await self._handle_new_receipts(receipts)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
res = await self.store.insert_receipt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
self.db_pool.simple_upsert_txn(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
2023-07-18 14:56:46,937 - synapse.federation.federation_server - 1470 - ERROR - PUT-331507 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
await handler(origin, content)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
await self._handle_new_receipts(receipts)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
res = await self.store.insert_receipt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
self.db_pool.simple_upsert_txn(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
2023-07-18 14:56:46,968 - synapse.federation.federation_server - 1470 - ERROR - PUT-331507 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
await handler(origin, content)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
await self._handle_new_receipts(receipts)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
res = await self.store.insert_receipt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
self.db_pool.simple_upsert_txn(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
I enabled INFO
log on the error log. Maybe this will give you more context?
2023-07-22 17:48:34,385 - synapse.storage.databases.main.event_push_actions - 1596 - INFO - rotate_notifs-284 - Rotating notifications, handling 0 rows
2023-07-22 17:48:34,387 - synapse.storage.databases.main.event_push_actions - 1684 - INFO - rotate_notifs-284 - Rotating notifications, deleted 0 push actions
2023-07-22 17:48:34,453 - synapse.util.caches.lrucache - 210 - INFO - LruCache._expire_old_entries-284 - Dropped 1 items from caches
2023-07-22 17:48:34,460 - synapse.http.matrixfederationclient - 690 - INFO - _maybe_retry_device_resync-284 - {GET-O-16064} [matrix.bootlin.com] Got response headers: 403 Forbidden
2023-07-22 17:48:34,460 - synapse.http.matrixfederationclient - 780 - WARNING - _maybe_retry_device_resync-284 - {GET-O-16064} [matrix.bootlin.com] Request failed: GET matrix-federation://matrix.bootlin.com/_matrix/federation/v1/user/devices/%406frzr%3Amatrix.bootlin.com: HttpResponseException('403: Forbidden')
2023-07-22 17:48:34,460 - synapse.handlers.device - 1279 - WARNING - _maybe_retry_device_resync-284 - Failed to handle device list update for @6frzr:matrix.bootlin.com: 403: Forbidden
2023-07-22 17:48:36,264 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9453 - Received txn 1689898786362 from midnightthoughts.space. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:36,269 - synapse.access.http.8008 - 461 - INFO - PUT-9453 - 65.21.116.72 - 8008 - {midnightthoughts.space} Processed request: 0.005sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.003sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1689898786362 HTTP/1.1" "Synapse/1.87.0" [0 dbevts]
2023-07-22 17:48:37,558 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9454 - Received txn 1683237755090 from chapo.chat. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:37,571 - synapse.access.http.8008 - 461 - INFO - GET-9450 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 12.653sec/0.001sec (0.000sec, 0.000sec) (0.001sec/0.001sec/1) 418B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898754_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:37,572 - synapse.access.http.8008 - 461 - INFO - PUT-9454 - 94.23.36.217 - 8008 - {chapo.chat} Processed request: 0.015sec/0.000sec (0.001sec, 0.000sec) (0.001sec/0.008sec/6) 11B 200 "PUT /_matrix/federation/v1/send/1683237755090 HTTP/1.1" "Synapse/1.79.0" [0 dbevts]
2023-07-22 17:48:37,749 - synapse.metrics._gc - 120 - INFO - sentinel - Collecting gc 1
2023-07-22 17:48:38,117 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9457 - Received txn 1689380252884 from cyber.bingo. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:38,127 - synapse.access.http.8008 - 461 - INFO - GET-9456 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 0.532sec/0.000sec (0.000sec, 0.000sec) (0.001sec/0.001sec/1) 416B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898755_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:38,128 - synapse.access.http.8008 - 461 - INFO - PUT-9457 - 188.34.180.209 - 8008 - {cyber.bingo} Processed request: 0.012sec/0.000sec (0.001sec, 0.000sec) (0.001sec/0.007sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1689380252884 HTTP/1.1" "Synapse/1.87.0" [0 dbevts]
2023-07-22 17:48:43,085 - synapse.crypto.keyring - 687 - INFO - PerspectivesKeyFetcher-359 - Requesting keys [_FetchKeyRequest(server_name='forkk.me', minimum_valid_until_ts=1690048123084, key_ids=['ed25519:a_pYEz'])] from notary server matrix.org
2023-07-22 17:48:43,235 - synapse.http.matrixfederationclient - 346 - INFO - PerspectivesKeyFetcher-359 - {POST-O-16069} [matrix.org] Completed request: 200 OK in 0.15 secs, got 436 bytes - POST matrix-federation://matrix.org/_matrix/key/v2/query
2023-07-22 17:48:43,240 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9460 - Received txn 1689998892540 from forkk.me. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:43,247 - synapse.access.http.8008 - 461 - INFO - GET-9459 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 5.094sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 414B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898756_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:43,248 - synapse.access.http.8008 - 461 - INFO - PUT-9460 - 45.56.76.228 - 8008 - {forkk.me} Processed request: 0.163sec/0.000sec (0.001sec, 0.001sec) (0.001sec/0.004sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1689998892540 HTTP/1.1" "Synapse/1.88.0" [0 dbevts]
2023-07-22 17:48:44,000 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9463 - Received txn 1689812551086 from hacklab.fi. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:44,008 - synapse.access.http.8008 - 461 - INFO - GET-9462 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 0.739sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 396B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898757_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:44,010 - synapse.access.http.8008 - 461 - INFO - PUT-9463 - 94.237.12.85 - 8008 - {hacklab.fi} Processed request: 0.011sec/0.000sec (0.000sec, 0.000sec) (0.001sec/0.006sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1689812551086 HTTP/1.1" "Synapse/1.88.0" [0 dbevts]
2023-07-22 17:48:45,967 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9466 - Received txn 1689869472115 from matrix.org. (PDUs: 0, EDUs: 17)
2023-07-22 17:48:45,974 - synapse.handlers.typing - 398 - INFO - PUT-9466 - Got typing update from @mikeymongol:matrix.org: {'room_id': '!BZVTUuEiNmRcbFeLeI:matrix.org', 'typing': True, 'user_id': '@mikeymongol:matrix.org'}
2023-07-22 17:48:45,979 - synapse.access.http.8008 - 461 - INFO - GET-9465 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 1.943sec/0.003sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 659B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898758_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:45,987 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
0/5
2023-07-22 17:48:45,991 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
0/5
2023-07-22 17:48:45,992 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
1/5
2023-07-22 17:48:45,992 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
0/5
2023-07-22 17:48:45,996 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5bf} could not serialize access due to concurrent update
0/5
2023-07-22 17:48:45,997 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
1/5
2023-07-22 17:48:45,997 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
2/5
2023-07-22 17:48:45,997 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
1/5
2023-07-22 17:48:46,002 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5bf} could not serialize access due to concurrent update
1/5
2023-07-22 17:48:46,002 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
2/5
2023-07-22 17:48:46,002 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5c9} could not serialize access due to concurrent update
0/5
2023-07-22 17:48:46,003 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5cf} could not serialize access due to concurrent update
0/5
2023-07-22 17:48:46,003 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
2/5
2023-07-22 17:48:46,003 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
3/5
2023-07-22 17:48:46,008 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
3/5
2023-07-22 17:48:46,008 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
4/5
2023-07-22 17:48:46,008 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5c9} could not serialize access due to concurrent update
1/5
2023-07-22 17:48:46,008 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
3/5
2023-07-22 17:48:46,009 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5cf} could not serialize access due to concurrent update
1/5
2023-07-22 17:48:46,011 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5d6} could not serialize access due to concurrent update
0/5
2023-07-22 17:48:46,012 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
4/5
2023-07-22 17:48:46,012 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
4/5
2023-07-22 17:48:46,015 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5cf} could not serialize access due to concurrent update
2/5
2023-07-22 17:48:46,015 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5d6} could not serialize access due to concurrent update
1/5
2023-07-22 17:48:46,016 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5dc} could not serialize access due to concurrent update
0/5
2023-07-22 17:48:46,016 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
5/5
2023-07-22 17:48:46,019 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5dc} could not serialize access due to concurrent update
1/5
2023-07-22 17:48:46,020 - synapse.federation.federation_server - 1470 - ERROR - PUT-9466 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
await handler(origin, content)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
await self._handle_new_receipts(receipts)
File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
res = await self.store.insert_receipt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
self.db_pool.simple_upsert_txn(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
2023-07-22 17:48:46,022 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5dc} could not serialize access due to concurrent update
2/5
2023-07-22 17:48:46,022 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5d6} could not serialize access due to concurrent update
2/5
2023-07-22 17:48:46,029 - synapse.access.http.8008 - 461 - INFO - GET-9468 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 0.020sec/0.001sec (0.001sec, 0.000sec) (0.001sec/0.010sec/3) 735B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898758_336_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:46,031 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5d6} could not serialize access due to concurrent update
3/5
2023-07-22 17:48:46,036 - synapse.access.http.8008 - 461 - INFO - PUT-9466 - 176.126.240.158 - 8008 - {matrix.org} Processed request: 0.070sec/0.000sec (0.027sec, 0.011sec) (0.062sec/0.379sec/76) 11B 200 "PUT /_matrix/federation/v1/send/1689869472115 HTTP/1.1" "Synapse/1.88.0 (b=matrix-org-hotfixes,b89e8d695d)" [0 dbevts]
2023-07-22 17:48:46,746 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9471 - Received txn 1689697704599 from gnxlxnxx.space. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:46,751 - synapse.access.http.8008 - 461 - INFO - PUT-9471 - 185.232.68.89 - 8008 - {gnxlxnxx.space} Processed request: 0.005sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.003sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1689697704599 HTTP/1.1" "Synapse/1.87.0" [0 dbevts]
2023-07-22 17:48:51,324 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9473 - Received txn 1690023590890 from envs.net. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:51,329 - synapse.access.http.8008 - 461 - INFO - PUT-9473 - 188.40.204.226 - 8008 - {envs.net} Processed request: 0.005sec/0.000sec (0.002sec, 0.000sec) (0.000sec/0.003sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1690023590890 HTTP/1.1" "Synapse/1.88.0" [0 dbevts]
2023-07-22 17:48:51,835 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9474 - Received txn 1689836865658 from kittenface.studio. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:51,840 - synapse.access.http.8008 - 461 - INFO - PUT-9474 - 62.63.215.235 - 8008 - {kittenface.studio} Processed request: 0.005sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.003sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1689836865658 HTTP/1.1" "Synapse/1.88.0" [0 dbevts]
2023-07-22 17:48:52,436 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9475 - Received txn 1686843884444 from neat.chat. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:52,445 - synapse.access.http.8008 - 461 - INFO - GET-9470 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 6.390sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 415B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898758_336_1760522_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:52,446 - synapse.access.http.8008 - 461 - INFO - PUT-9475 - 5.161.59.30 - 8008 - {neat.chat} Processed request: 0.011sec/0.000sec (0.002sec, 0.000sec) (0.001sec/0.006sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1686843884444 HTTP/1.1" "Synapse/1.85.2" [0 dbevts]
2023-07-22 17:48:58,510 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9478 - Received txn 1689570710451 from discuss.online. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:58,513 - synapse.access.http.8008 - 461 - INFO - PUT-9478 - 138.197.80.206 - 8008 - {discuss.online} Processed request: 0.003sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.001sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1689570710451 HTTP/1.1" "Synapse/1.87.0" [0 dbevts]
2023-07-22 17:49:04,264 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9479 - Received txn 1685925939747 from finallycoffee.eu. (PDUs: 0, EDUs: 1)
2023-07-22 17:49:04,273 - synapse.access.http.8008 - 461 - INFO - GET-9477 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 11.803sec/0.001sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 408B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898759_336_1760522_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:49:04,275 - synapse.access.http.8008 - 461 - INFO - PUT-9479 - 78.94.116.221 - 8008 - {finallycoffee.eu} Processed request: 0.011sec/0.000sec (0.001sec, 0.000sec) (0.001sec/0.006sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1685925939747 HTTP/1.1" "Synapse/1.84.1" [0 dbevts]
2023-07-22 17:49:04,382 - synapse.storage.databases.main.event_push_actions - 1306 - INFO - rotate_notifs-285 - Rotating notifications
2023-07-22 17:49:04,383 - synapse.storage.databases.main.event_push_actions - 1510 - INFO - rotate_notifs-285 - Rotating notifications up to: 8272075
2023-07-22 17:49:04,384 - synapse.storage.databases.main.event_push_actions - 1596 - INFO - rotate_notifs-285 - Rotating notifications, handling 0 rows
2023-07-22 17:49:04,386 - synapse.storage.databases.main.event_push_actions - 1684 - INFO - rotate_notifs-285 - Rotating notifications, deleted 0 push actions
2023-07-22 17:49:04,446 - synapse.handlers.presence - 795 - INFO - persist_presence_changes-141 - Persisting 10 unpersisted presence updates
2023-07-22 17:49:04,448 - synapse.federation.sender - 979 - INFO - wake_destinations_needing_catchup-142 - Destination gaw.social has outstanding catch-up, waking up.
2023-07-22 17:49:04,449 - synapse.federation.sender.per_destination_queue - 486 - INFO - federation_transaction_transmission_loop-11754 - Catching up destination gaw.social with 1 PDUs
2023-07-22 17:49:04,450 - synapse.federation.sender.per_destination_queue - 568 - INFO - federation_transaction_transmission_loop-11754 - Catching up rooms to gaw.social: '!gWiVYCURkxIapWZaGy:ipfs.io'
2023-07-22 17:49:04,450 - synapse.federation.sender.transaction_manager - 121 - INFO - federation_transaction_transmission_loop-11754 - TX [gaw.social] {1690039575215} Sending transaction [1690039575215], (PDUs: 1, EDUs: 0)
2023-07-22 17:49:04,452 - synapse.util.caches.lrucache - 210 - INFO - LruCache._expire_old_entries-285 - Dropped 1 items from caches
Same error here, with the same setup. Let me know if I can help with logs or anything.
I've got a similar issue on my end. If it helps, I've also got the sliding sync proxy running.
Same issue.
Disabling siding sync proxy doesn’t help.
In fact, when I first knew about this error, I didn’t have any siding sync proxy running.
https://sentry.tools.element.io/organizations/element/issues/69241 looks like an instance of this.
Looks like the error comes from https://github.com/matrix-org/synapse/blob/8c3bcea2da4939e21a99f72d6c3995186bc4b80d/synapse/federation/federation_server.py#L1481
Would need some investigation to work out what the database is unhappy about.
Got the same issue. More or less the same setup. Synapse 1.89.0 running in docker with a PostgreSQL 15 database on another virtual machine. If you need logs or tests run let me know.
Please check again if this still happens after 1.93.0 has been released. A fix is included that may stop this instance occurring.
Description
Despite that issue: https://github.com/matrix-org/synapse/issues/3234 is closed. Is is not resolved for me.
I still suffer from errors: "ERROR - PUT-58315 - Failed to handle edu 'm.receipt'". See log output below, for more info.
Steps to reproduce
Homeserver
matrix.melroy.org
Synapse Version
{"server_version":"1.87.0","python_version":"3.11.4"}
Installation Method
Docker (matrixdotorg/synapse)
Database
Postgresql V14
Workers
Single process
Platform
Ubuntu 22.04 (running Docker on top of that)
Configuration
Standard Matrix config
Relevant log output
Anything else that would be useful to know?
No response