search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.82k stars 2.13k forks source link

Upgrade from 1.76.0 to 1.95.1 results in synapse.storage.background_updates: [background_updates-0] Error doing update: 'int' object is not iterable #16591

Closed Artiom-M closed 12 months ago

Artiom-M commented 1 year ago

Description

After standard upgrade with pip service starts and apparently working, but log is full of errors about Error doing update: 'int' object is not iterable matrix.log

Homeserver

self hosted

Synapse Version

{"server_version":"1.95.1"}

Installation Method

PIP

Database

PostgreSQL single server

Workers

Single process

Platform

Oracle linux 8

Configuration

No response

Relevant log output

Nov 01 05:06:58  matrix-synapse[3419535]: 2023-11-01 05:06:58,673 - synapse.storage.background_updates - 412 - ERROR - background_updates-0 - Error doing update: 'int' object is not iterable
Nov 01 05:06:58  matrix-synapse[3419535]: Traceback (most recent call last):
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/background_updates.py", line 409, in run_background_updates
Nov 01 05:06:58  matrix-synapse[3419535]:     result = await self.do_next_background_update(sleep)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/background_updates.py", line 540, in do_next_background_update
Nov 01 05:06:58  matrix-synapse[3419535]:     await self._do_background_update(desired_duration_ms)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/background_updates.py", line 583, in _do_background_update
Nov 01 05:06:58  matrix-synapse[3419535]:     items_updated = await update_handler(progress, batch_size)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/databases/main/pusher.py", line 635, in _set_device_id_for_pushers
Nov 01 05:06:58  matrix-synapse[3419535]:     nb_processed = await self.db_pool.runInteraction(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 934, in runInteraction
Nov 01 05:06:58  matrix-synapse[3419535]:     return await delay_cancellation(_runInteraction())
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
Nov 01 05:06:58  matrix-synapse[3419535]:     result = context.run(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
Nov 01 05:06:58  matrix-synapse[3419535]:     return g.throw(self.type, self.value, self.tb)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 900, in _runInteraction
Nov 01 05:06:58  matrix-synapse[3419535]:     result = await self.runWithConnection(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 1029, in runWithConnection
Nov 01 05:06:58  matrix-synapse[3419535]:     return await make_deferred_yieldable(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/threadpool.py", line 244, in inContext
Nov 01 05:06:58  matrix-synapse[3419535]:     result = inContext.theWork()  # type: ignore[attr-defined]
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
Nov 01 05:06:58  matrix-synapse[3419535]:     inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/context.py", line 117, in callWithContext
Nov 01 05:06:58  matrix-synapse[3419535]:     return self.currentContext().callWithContext(ctx, func, *args, **kw)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/context.py", line 82, in callWithContext
Nov 01 05:06:58  matrix-synapse[3419535]:     return func(*args, **kw)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
Nov 01 05:06:58  matrix-synapse[3419535]:     result = func(conn, *args, **kw)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 1022, in inner_func
Nov 01 05:06:58  matrix-synapse[3419535]:     return func(db_conn, *args, **kwargs)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 762, in new_transaction
Nov 01 05:06:58  matrix-synapse[3419535]:     r = func(cursor, *args, **kwargs)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/databases/main/pusher.py", line 614, in set_device_id_for_pushers_txn
Nov 01 05:06:58  matrix-synapse[3419535]:     self.db_pool.simple_update_many_txn(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 2078, in simple_update_many_txn
Nov 01 05:06:58  matrix-synapse[3419535]:     args = [tuple(x) + tuple(y) for x, y in zip(value_values, key_values)]
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 2078, in <listcomp>
Nov 01 05:06:58  matrix-synapse[3419535]:     args = [tuple(x) + tuple(y) for x, y in zip(value_values, key_values)]
Nov 01 05:06:58  matrix-synapse[3419535]: TypeError: 'int' object is not iterable

Anything else that would be useful to know?

No response

erikjohnston commented 1 year ago

Hi @Artiom-M, thanks, this indeed looks like a bug. It's in a background update though so shouldn't affect day-to-day operation beyond spamming the logs with errors. Usually I'd suggest downgrading again, but since v1.95.1 is a security release you might want to stay on v1.95.1.

The problem is:

https://github.com/matrix-org/synapse/blob/a11511954a58975d2e5400257a0cecfd27413447/synapse/storage/databases/main/pusher.py#L617-L618

That second line should be a list of tuples, i.e. [(row[0],) ...]

Broke in #16431.

cc @clokep.

clokep commented 12 months ago

16594 should fix this.

Artiom-M commented 12 months ago

As this is apparently not breaking anything for me, I will stay with 1.95.1 waiting for release. Thank you.

P.S. Actually I have just applied the fix manually to the file.