search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.83k stars 2.12k forks source link

API endpoint `/_synapse/admin/v1/statistics/database/rooms` throwing Python error #15821

Open daedric7 opened 1 year ago

daedric7 commented 1 year ago

Description

When attempting to use the API endpoint /_synapse/admin/v1/statistics/database/rooms via curl, a HTTP 500 is received while Synapse throws Python errors in the log.

Steps to reproduce

  1. curl -XGET --header "Authorization: Bearer TOKENTOKEN" http://synapse-server/_synapse/admin/v1/statistics/database/rooms

Homeserver

aguiarvieira.pt

Synapse Version

1.86.0

Installation Method

Docker (matrixdotorg/synapse)

Database

PostgreSQL

Workers

Multiple workers

Platform

Oracle Arm Free with Docker

Configuration

Presence: yes Retention: no Synapse modules: shared_secret_authenticator.SharedSecretAuthProvider

Relevant log output

matrix-matrix-synapse-1  | 2023-06-23 10:33:19,794 - synapse.http.server - 133 - ERROR - GET-3694 - Failed handle request via 'LargestRoomsStatistics': <XForwardedForRequest at 0xffff61ac8750 method='GET' uri='/_synapse/admin/v1/statistics/database/rooms' clientproto='HTTP/1.1' site='8008'>
matrix-matrix-synapse-1  | Traceback (most recent call last):
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 319, in _async_render_wrapper
matrix-matrix-synapse-1  |     callback_return = await self._async_render(request)
matrix-matrix-synapse-1  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 529, in _async_render
matrix-matrix-synapse-1  |     callback_return = await raw_callback_return
matrix-matrix-synapse-1  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/rest/admin/statistics.py", line 133, in on_GET
matrix-matrix-synapse-1  |     room_sizes = await self.stats_controller.get_room_db_size_estimate()
matrix-matrix-synapse-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/stats.py", line 93, in get_room_db_size_estimate
matrix-matrix-synapse-1  |     await self.stores.main.db_pool.runInteraction(
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
matrix-matrix-synapse-1  |     return await delay_cancellation(_runInteraction())
matrix-matrix-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
matrix-matrix-synapse-1  |     result = context.run(
matrix-matrix-synapse-1  |              ^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
matrix-matrix-synapse-1  |     return g.throw(self.type, self.value, self.tb)
matrix-matrix-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
matrix-matrix-synapse-1  |     result = await self.runWithConnection(
matrix-matrix-synapse-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
matrix-matrix-synapse-1  |     return await make_deferred_yieldable(
matrix-matrix-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
matrix-matrix-synapse-1  |     result = inContext.theWork()  # type: ignore[attr-defined]
matrix-matrix-synapse-1  |              ^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
matrix-matrix-synapse-1  |     inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
matrix-matrix-synapse-1  |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
matrix-matrix-synapse-1  |     return self.currentContext().callWithContext(ctx, func, *args, **kw)
matrix-matrix-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
matrix-matrix-synapse-1  |     return func(*args, **kw)
matrix-matrix-synapse-1  |            ^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
matrix-matrix-synapse-1  |     result = func(conn, *args, **kw)
matrix-matrix-synapse-1  |              ^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
matrix-matrix-synapse-1  |     return func(db_conn, *args, **kwargs)
matrix-matrix-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
matrix-matrix-synapse-1  |     r = func(cursor, *args, **kwargs)
matrix-matrix-synapse-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/stats.py", line 90, in get_room_db_size_estimate_txn
matrix-matrix-synapse-1  |     for room_id, freq in zip(vals, freqs):
matrix-matrix-synapse-1  |                          ^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1  | TypeError: 'NoneType' object is not iterable

Anything else that would be useful to know?

No response

DMRobertson commented 1 year ago

Corresponding API docs are https://matrix-org.github.io/synapse/latest/admin_api/statistics.html#get-largest-rooms-by-size-in-database

MadLittleMods commented 1 year ago

Looks like there is a couple edge cases we should take into account in the stats code.

For example, just running the SQL query locally, I see that tables can return empty or just be completely omitted even after a ANALYZE;

SELECT
    tablename, most_common_vals::TEXT::TEXT[], most_common_freqs::TEXT::NUMERIC[]
FROM pg_stats
WHERE
  tablename IN ('event_json', 'events', 'event_search', 'event_edges', 'event_push_actions', 'stream_ordering_to_exterm')
  and attname = 'room_id';
         tablename         |                                     most_common_vals                                      |   most_common_freqs
---------------------------+-------------------------------------------------------------------------------------------+------------------------
 event_json                | {!KQuSGAhSwdtQiqBxGw:my.synapse.linux.server,!UVksaBnsYTjHqLmOqz:my.synapse.linux.server} | {0.75,0.25}
 event_search              | {!KQuSGAhSwdtQiqBxGw:my.synapse.linux.server,!UVksaBnsYTjHqLmOqz:my.synapse.linux.server} | {0.9,0.1}
 stream_ordering_to_exterm | {!KQuSGAhSwdtQiqBxGw:my.synapse.linux.server,!UVksaBnsYTjHqLmOqz:my.synapse.linux.server} | {0.8636364,0.13636364}
 events                    | {!KQuSGAhSwdtQiqBxGw:my.synapse.linux.server,!UVksaBnsYTjHqLmOqz:my.synapse.linux.server} | {0.75,0.25}
 event_edges               |                                                                                           |
(5 rows)