When a worker cannot connect to another the log messages are unhelpful

Description

Media worker tried to connect a wrong IP for federation sender. The log contained various crazy stuff:

remote server refused connection
remote server gave 502 error (bad gateway)

Even enabled DEBUG didn't reveal anything obvious. Lot of stracing and tcpdumps revealed that media worker wanted to connect a wrong IP, which in turn made me find the only hint the log contained, which is

2023-09-11 22:19:45,884 - synapse.http.proxyagent - 313 - DEBUG - GET-42- Requesting matrix-federation://norge.chat/_matrix/media/r0/download/norge.chat/FWVlbNUArgokLhAMRyhvEDON?allow_remote=false via <_RandomSampleEndpoints endpoints=[<HostnameEndpoint BADIP:28311>, <HostnameEndpoint BADIP:28312>]>

Fixing the ip resolved the problem, but the log have not helped at all. Neither told the real problem nor was it revealing that it has logged fake errors if they were given from the remote server and not our own worker [replication] connection.

Steps to reproduce

use bad ip for federation sender worker in the config of the media worker
try to retrieve media
observe the lack of real errors and plenty of fake ones

Homeserver

grin.hu

Synapse Version

1.90.0-1

Installation Method

Other (please mention below)

Database

psql

Workers

Multiple workers

Platform

Debian/bookworm. Installed from debian pkg, not that it matters.

Configuration

N/A

Relevant log output

2023-09-11 18:02:57,960 - synapse.http.matrixfederationclient - 758 - INFO - GET-19848- {GET-O-6479} [infeeeee.duckdns.org] Request failed: GET matrix-federation://infeeeee.duckdns.org/_matrix/media/r0/download/infeeeee.duckdns.org/PuraUIczWXOjSUBjeTQJwPpf?allow_remote=false: ConnectionRefusedError('Connection refused')

2023-09-11 18:40:33,325 - synapse.http.server - 118 - INFO - GET-19962- <SynapseRequest at 0x7f9463db5600 method='GET' uri='/_matrix/media/r0/download/infeeeee.duckdns.org/PuraUIczWXOjSUBjeTQJwPpf' clientproto='HTTP/1.1' site='8085'> SynapseError: 502 - Failed to fetch remote media

Anything else that would be useful to know?

I don't think so.

matrix-org / synapse