Real-world 0.17.4 Lemmy servers are failing with federation - error logs show 'Header is expired'

[RocketDerp]

From what i have learned, Lemmy is creating http connections that are short-lived (only 10 minutes?) before they expire or some similar scheme related to signing/encryption?

In real-word servers, I'm seeing logging of:

WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Header is expired
  0: lemmy_server::root_span_builder::HTTP request
            with http.method=POST http.scheme="http" http.host=mylemmyinstance.com http.target=/inbox otel.kind="server" request_id=453c8a92-7bb5-4b7e-a4ad-212e91167d4e http.status_code=400 otel.status_code="OK"
         at src/root_span_builder.rs:16
 LemmyError { message: None, inner: Header is expired, context: "SpanTrace" }

Because of proxying by nginx, this message does not give any hint who the sender is. IF the clocks are not set right on one of the peer servers, it could be a major problem and you would have to start logging on the firewall or something to find any hint of who the sender is.

I did tinker around with the code and I at least was able to get Lemmy to log the IP address of the remote server by adding realip_remote_addr to the tracing:

+++ b/src/root_span_builder.rs
@@ -18,6 +18,7 @@ impl RootSpanBuilder for QuieterRootSpanBuilder {
         http.method= %request.method(),
         http.scheme = request.connection_info().scheme(),
         http.host = %request.connection_info().host(),
+        http.realip_remote_addr = request.connection_info().realip_remote_addr(),

MOST IMPORTANT TO ME: is we need to get the word out to the major Lemmy instances to be looking for this message in their error log. Is this why we are seeing significant failures to replicate data between servers? Issues: https://github.com/LemmyNet/lemmy/issues/3101 and https://github.com/LemmyNet/lemmy/issues/3203

[Nutomic]

HTTP signatures are currently valid for only 5 minutes. Will make a PR to change that.