search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.82k stars 2.13k forks source link

Exception running callback for LaterGauge(synapse_rate_limit_reject_affected_hosts) #14319

Open DMRobertson opened 2 years ago

DMRobertson commented 2 years ago

Like #12078 and #12079.

https://sentry.tools.element.io/organizations/element/issues/33651/?project=2&query=is%3Aunresolved

2022-10-28 10:58:54,711 - synapse.metrics - 100 - ERROR - sentinel - Exception running callback for LaterGauge(synapse_rate_limit_reject_affected_hosts)
Capture point (most recent call last):
  File "/home/synapse/pyston_2.3.3/lib/python3.8-pyston2.3/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/home/synapse/pyston_2.3.3/lib/python3.8-pyston2.3/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/synapse/env-pyston-poetry/lib/python3.8-pyston2.3/site-packages/sentry_sdk/integrations/threading.py", line 67, in run
    return old_run_func(self, *a, **kw)
  File "/home/synapse/pyston_2.3.3/lib/python3.8-pyston2.3/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/synapse/pyston_2.3.3/lib/python3.8-pyston2.3/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/home/synapse/pyston_2.3.3/lib/python3.8-pyston2.3/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/synapse/pyston_2.3.3/lib/python3.8-pyston2.3/socketserver.py", line 747, in __init__
    self.handle()
  File "/home/synapse/pyston_2.3.3/lib/python3.8-pyston2.3/wsgiref/simple_server.py", line 134, in handle
    handler.run(self.server.get_app())
  File "/home/synapse/pyston_2.3.3/lib/python3.8-pyston2.3/wsgiref/handlers.py", line 137, in run
    self.result = application(self.environ, self.start_response)
  File "/home/synapse/env-pyston-poetry/lib/python3.8-pyston2.3/site-packages/prometheus_client/exposition.py", line 127, in prometheus_app
    status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
  File "/home/synapse/env-pyston-poetry/lib/python3.8-pyston2.3/site-packages/prometheus_client/exposition.py", line 103, in _bake_output
    output = formatter(registry)
  File "/home/synapse/env-pyston-poetry/lib/python3.8-pyston2.3/site-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest
    for metric in registry.collect():
  File "/home/synapse/src/synapse/metrics/__init__.py", line 70, in collect
    for metric in REGISTRY.collect():
  File "/home/synapse/env-pyston-poetry/lib/python3.8-pyston2.3/site-packages/prometheus_client/registry.py", line 97, in collect
    yield from collector.collect()
Traceback (most recent call last):
  File "/home/synapse/src/synapse/metrics/__init__.py", line 98, in collect
    calls = self.caller()
  File "/home/synapse/src/synapse/util/ratelimitutils.py", line 135, in <lambda>
    lambda: _get_counts_from_rate_limiter_instance(
  File "/home/synapse/src/synapse/util/ratelimitutils.py", line 112, in _get_counts_from_rate_limiter_instance
    counts[key] = count_func(rate_limiter_instance)
  File "/home/synapse/src/synapse/util/ratelimitutils.py", line 136, in <lambda>
    lambda rate_limiter_instance: sum(
  File "/home/synapse/src/synapse/util/ratelimitutils.py", line 136, in <genexpr>
    lambda rate_limiter_instance: sum(
RuntimeError: dictionary changed size during iteration
DMRobertson commented 2 years ago

Relevant:

https://github.com/matrix-org/synapse/blob/be4250c7a888e314e361df42042bfa344ab65d55/synapse/metrics/__init__.py#L93-L110

DMRobertson commented 2 years ago

Like #12078 and #12079.

Are these three instances of the same underlying problem?