Synapse v1.38 throwing out multiple exceptions when trying to join #offtopic:matrix.org

Description

I am trying to join #offtopic:matrix.org on a 4 worker synapse v1.38 setup (2 are outbound federation workers and the other 2 workers are inbound federation workers). Each worker has around 2-4GB of RAM allocated, and the main synapse process has up to 16GB of RAM to eat up, for its own sake. Synapse by itself, is using around 3-4GB of RAM... not sure if that is normal or not.

Steps to reproduce

Join #offtopic:matrix.org with the described above setup, with PostgreSQL and redis running on the same PC that is hosting the main synapse process.

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1445, in _inlineCallbacks
    result = current_context.run(g.send, result)
StopIteration: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1445, in _inlineCallbacks
    result = current_context.run(g.send, result)
StopIteration: {'ed25519:a_RMgP': FetchKeyResult(verify_key=<nacl.signing.VerifyKey object at 0x7eff4d085b70>, valid_until_ts=1625822662669)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_base.py", line 147, in _check_sigs_on_pdu
    pdu.origin_server_ts if room_version.enforce_key_validity else 0,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/crypto/keyring.py", line 246, in verify_event_for_server
    validity_time,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/crypto/keyring.py", line 292, in process_request
    verify_key,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/signedjson/sign.py", line 126, in verify_signed_json
    message = encode_canonical_json(json_object_copy)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/canonicaljson.py", line 73, in encode_canonical_json
    s = _canonical_encoder.encode(json_object)
  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/canonicaljson.py", line 31, in _default
    "Object of type %s is not JSON serializable" % obj.__class__.__name__
TypeError: Object of type Decimal is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_client.py", line 545, in _try_destination_list
    return await callback(destination)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_client.py", line 753, in send_request
    _execute, itertools.chain(state, auth_chain), 10000
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1445, in _inlineCallbacks
    result = current_context.run(g.send, result)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/util/async_helpers.py", line 187, in _concurrently_execute_inner
    await maybe_awaitable(func(value))
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_client.py", line 746, in _execute
    room_version=room_version,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_client.py", line 432, in _check_sigs_and_hash_and_fetch_one
    res = await self._check_sigs_and_hash(room_version, pdu)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_base.py", line 56, in _check_sigs_and_hash
    await _check_sigs_on_pdu(self.keyring, room_version, pdu)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_base.py", line 151, in _check_sigs_on_pdu
    pdu.event_id,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/events/__init__.py", line 492, in event_id
    compute_event_reference_hash(self)[1], urlsafe=True
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/crypto/event_signing.py", line 121, in compute_event_reference_hash
    event_json_bytes = encode_canonical_json(event_dict)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/canonicaljson.py", line 73, in encode_canonical_json
    s = _canonical_encoder.encode(json_object)
  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/canonicaljson.py", line 31, in _default
    "Object of type %s is not JSON serializable" % obj.__class__.__name__
TypeError: Object of type Decimal is not JSON serializable
2021-07-17 01:52:22,591 - synapse.federation.federation_base - 80 - INFO - POST-57489 - Event $7Ua7nFtvFMG1zBNnweFPF-7gf2YuChhkvnRCUW5MZII seems to have been redacted; using our redacted copy
2021-07-17 01:52:22,711 - synapse.replication.http.federation - 172 - INFO - POST-57522 - Got 'm.presence' edu from gnu.fun
2021-07-17 01:52:22,711 - synapse.access.http.9093 - 410 - INFO - POST-57522 - 10.1.0.9 - 9093 - {None} Processed request: 0.001sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 4B 200 "POST /_synapse/replication/fed_send_edu/m.presence/SAmINxroDv HTTP/1.1" "Synapse/1.38.0" [0 dbevts]
2021-07-17 01:52:22,711 - synapse.federation.federation_client - 576 - WARNING - POST-57489 - Failed to send_join via eggy.cc
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1445, in _inlineCallbacks
    result = current_context.run(g.send, result)
StopIteration: DestinationRetryTimings(failure_ts=1626292959319, retry_last_ts=1626329203587, retry_interval=164995183)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_client.py", line 545, in _try_destination_list
    return await callback(destination)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_client.py", line 704, in send_request
    response = await self._do_send_join(room_version, destination, pdu)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/federation_client.py", line 806, in _do_send_join
    content=pdu.get_pdu_json(time_now),
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/federation/transport/client.py", line 286, in send_join_v2
    max_response_size=MAX_RESPONSE_SIZE_SEND_JOIN,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/http/matrixfederationclient.py", line 847, in put_json
    timeout=timeout,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/http/matrixfederationclient.py", line 381, in _send_request_with_optional_trailing_slash
    response = await self._send_request(request, **send_request_args)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/http/matrixfederationclient.py", line 474, in _send_request
    ignore_backoff=ignore_backoff,
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/util/retryutils.py", line 95, in get_retry_limiter
    destination=destination,
synapse.util.retryutils.NotRetryingDestination: Not retrying server eggy.cc.

Version information

Homeserver: I would prefer not to reveal it.
apt
Platform: Debian v10.
See first paragraph. The VMs are KVM-based VMs, on a hypervisor that has around 16GB of RAM available. Everything is being wireguarded to a VPS, to which, it is then forwarded back to the real servers.

matrix-org / synapse