Relay not forwarding to Sentry when behind nginx proxy

[DieBambusleitung]

I'm trying to use the Relay behind an nginx reverse proxy since it is recommended in the operating guidelines and for usage with ssl/https.

Unfortunately the relay seems not to forward the errors sent to it while running behind the proxy. The Relay works when sending errors directly to it without the proxy in front. I read the documentation but couldn't find anything useful except that the headers have to be set correctly, which I did in nginx configuraton. (see https://docs.sentry.io/product/relay/operating-guidelines/#request-routing)

I think I am missing something crucial, but am not able to fix it and seeking for assistance.

Detailed information for reproduction:

System Information: Sentry Relay Version: 23.8.0 Runtime: Docker OS: Debian 11 bullseye

Docker-Compose File:

version: "3"
services:
  nginx:
    image: nginx
    volumes:
      - ./nginx:/etc/nginx
      - ./cert:/ssl
    ports:
      - 80:80
      - 443:443
    environment:
      TZ: "Europe/Berlin"
  relay:
    image: getsentry/relay:latest
    command: ["run"]
    volumes:
      -  ./config/:/work/.relay/:z
    environment:
      TZ: "Europe/Berlin"

Sentry Relay Config:

I also tried using the Relay in proxy mode, which also had no impact on the result.

relay:
  mode: managed
  upstream: https://<SENTRY_FQDN>/
  host: 0.0.0.0
  port: 3000
logging:
  level: trace
sentry:
  enable: true

Nginx Configuration:

worker_processes 1;
events { worker_connections 1024; }
http {
    server {
        listen 80;
        server_name <HOST>;

        location / {
            return 301 https://$host$request_uri;
        }
    }
    server {
        listen 443 ssl;
        server_name <HOST>;

        underscores_in_headers on;

        ssl_certificate <PATH_TO_CERT>;
        ssl_certificate_key <PATH_TO_KEY>;

        location / {
           proxy_pass   http://relay:3000;
           proxy_set_header Host $host;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Sentry-Auth $http_x_sentry_auth;
           proxy_set_header Content-Type $http_content_type;
           proxy_set_header User-Agent $http_user_agent;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header sentry-trace $http_sentry_trace;
           proxy_set_header baggage $http_baggage;
           proxy_set_header Access-Control-Allow-Headers "sentry-trace, baggage";
        }
    }
}

I attached a log from Docker for more information. docker.log

[iker-barriocanal]

@getsentry/ops could probably provide good insights on what could be missing.

[iker-barriocanal]

Hi @DieBambusleitung, getting back to this. Is Relay getting events when you add the proxy in front of it? Relay should emit logs in this case.

[DieBambusleitung]

Hi @iker-barriocanal. Yes it does, please have a look at the attached docker.log file. The Relay receives the events and after that it’s logging with „ sending envelope to sentry endpoint“. My log level is „trace“.

[iker-barriocanal]

Based on sending envelope to sentry endpoint logs, it seems Relay is working as expected. Is there another proxy that might interfere with outgoing Relay requests?

[DieBambusleitung]

No there is no additional proxy. It works fine with plain http directly to the Relay with the same endpoint configuration, that’s what makes me confused. I also tried the „proxy“ mode of the Relay, so that every request gets forwarded. Unfortunately this had no impact on the results.

[iker-barriocanal]

Does the project have "allowed domains" configured? This configuration is available in project settings > general settings > allowed domains ("client security" section). Events may be dropped if Origin or Referer headers are missing.

[DieBambusleitung]

I will confirm this on Monday. What I can tell is, that I don't have any dropped or blocked events in the Sentry Stats.

About the headers: I "tcpdumped"/caputred both, headers with proxy infront and headers without proxy infront. Unfortunately the logs are on my working device so I will publish them as soon as I can.

It's very likely that my mistake has something to do with the headers, but I made sure that I forward the headers mentioned in the operation guidelines of the documentation. (see: https://docs.sentry.io/product/relay/operating-guidelines/#request-routing)

[jjbayer]

@DieBambusleitung are you connecting to sentry.io, or to a self-hosted instance? Asking because the ID in your logs does not look like a sentry.io project ID.

[DieBambusleitung]

Good look there. It's indeed a self-hosted sentry.

[jjbayer]

@DieBambusleitung the operating guidelines you linked are for putting a Relay instance between your software and sentry.io. The default self-hosted set already has a Relay instance and nginx setup out of the box. Does that help you?

[DieBambusleitung]

@jjbayer Thanks for the quick answer! I will try the provided nginx configuration on Wednesday. Although the self-hosted version has a Relay already infront of it, we want to use a Relay instance for applications on our customers sites.

[DieBambusleitung]

@jjbayer That fixed the problem! Thank you very much. I'm not quite sure if I have missed something in the docs, but I could not find this information there. Would be great if this could be added to the docs.

[jjbayer]

@DieBambusleitung what change did you apply exactly to fix the problem? Happy to add a note to the docs about it. BTW our docs are open source, so contributions are always welcome!

[DieBambusleitung]

@jjbayer Thanks for letting me know! I changed the provided nginx config so that it fits my environment.

I have to reopen because one of our JBoss runners still is not able to send errors. This behavior still just appears when using Relay behind the nginx. I attached the Relay logs. It's working fine with the sentry-cli.

relay-censored.log

What confuses me here again are the url query parameters that are attached to the events sent from the JBoss. Is the Relay able to handle those?

About the JBoss configuration: I will talk to the corresponding colleague about more information. What I can tell is that the DSNs have the following format: SET SENTRY_BACKEND_DSN=https://<DSN>@<RELAY-FQDN>/68

Looking forward to hear from you!

Does the project have "allowed domains" configured? This configuration is available in project settings > general settings > allowed domains ("client security" section). Events may be dropped if Origin or Referer headers are missing.

@iker-barriocanal I can confirm that all domains are allowed in our configuration and therefore won't get dropped. I also can't see any dropped Events in the stats of our "mother sentry"

EDIT:

We tested the exact same setup but with treafik instead of nginx and it works like a charm. Still confused what is wrong here.

[jjbayer]

@DieBambusleitung

from your Relay logs it seems that they receive and forward the data just fine:

Envelope received from the project you set up with SENTRY_BACKEND_DSN:

2023-10-04T09:35:57.619969Z TRACE request{method=POST uri=/api/68/envelope/ version=HTTP/1.1}: relay_server::endpoints::common: queueing envelope

then

2023-10-04T09:35:57.746832Z TRACE relay_server::actors::envelopes: sending envelope to sentry endpoint

Do you also have logs of the internal Relay of you self-hosted setup (docker-compose logs relay)?

[DieBambusleitung]

@jjbayer the log entry you mentioned is related to a test message via sentry-cli, which gets forwarded correctly. Please have a look at the other messages, where url query parameters are attached to the URI. These Events wont get forwarded to sentry.

Unfortunately I dont have acces to the internal Relay logs of the self-hosted sentry, but I will request them from my corresponding colleague.

[jjbayer]

@DieBambusleitung the query parameters look like the normal parameters set by the javascript SDK (see https://develop.sentry.dev/sdk/overview/#authentication) so they should be fine. sentry-cli seems to use header authentication instead.

Regardless of the project ID (68, 69, 75), the 17 occurrences of sending envelope to sentry endpoint in your log match the 17 occurrences of started processing request. So I suspect the events are dropped at a later stage, which is why logs from the inner Relay would be interesting.

[DieBambusleitung]

@jjbayer i am very happy but also sorry to say that it turned out to be a Layer 8 Problem. The Truststore of the JBoss was faulty and caused an error on the JBoss site, which leaded to not sending the error to the Relay.

Thank you very much for your responsiveness and time! @jjbayer @iker-barriocanal

Case closed with a facepalm