Question | Symstore proxy with self-hosted

[thorntonryan]

Question

Hello! Super cool project and I'm really excited about exploring to see if we can use sentry+symbolicator as a symbol server proxy for a native C++/Qt family of applications / libraries.

However, I'm struggling to put the pieces together to leverage symstore_proxy alongside a pretty stock sentry-self hosted install.

I've got sentry up and running and can upload debug symbols, but I'm struggling to pull those symbols down via Visual Studio.

I've tried pointing Visual Studio at both:

• http://127.0.0.1:SENTRY_BIND/proxy
• http://127.0.0.1:SENTRY_BIND/symbols

But neither is working. The symbols that show up are actually HTML (a redirect of sorts), so something isn't quite configured properly (apparently).

Any suggestions?

It looks like the standalone symbolicator exposes port 3021: https://github.com/getsentry/symbolicator/blob/9373cb5dc127f943be56ebecd7ca35a6340a56e2/docs/api/proxy.md?plain=1#L15 https://github.com/getsentry/symbolicator/blob/9373cb5dc127f943be56ebecd7ca35a6340a56e2/Dockerfile#L71

Do I need to do that also inside my docker-compose file somehow?

Take care, Ryan

Note, feel free to close if I should have asked this question in the self-hosted repo

Extras

I've got symbolicator setup with the following:

config.yml

# See: https://getsentry.github.io/symbolicator/#configuration
cache_dir: "/data"
bind: "0.0.0.0:3021"
logging:
  level: "warn"
metrics:
  statsd: null
sentry_dsn: null # TODO: Automatically fill this with the internal project DSN

symstore_proxy: true

# Configure some default symbol sources:
sources:
  # More public symbol servers be found here:
  # https://github.com/getsentry/sentry/blob/06265eed51c7f884b571bf55fe05312591ea146b/src/sentry/conf/server.py#L1894
  - id: sentry:microsoft
    type: http
    url: https://msdl.microsoft.com/download/symbols/
    layout: { type: "symstore" }
    filters: { filetypes: ["pe", "pdb", "portablepdb"] }
    is_public: true

    # In order to download debug files on demand directly from upstream sentry:
  - id: sentry:project
    type: sentry
    # Put the correct organization and project slug in the URL:
    url: http://web/api/0/projects/sentry/foo/files/dsyms/
    # You can use a personal auth token here that needs to have at least `project:read` permissions.
    # Create a new token here: https://sentry.io/settings/account/api/auth-tokens/
    token: <foo_token>

  # Sentry employees can find other non-public symbol servers here:
  # https://github.com/getsentry/getsentry/blob/54c52e5d844be11f7dda9c76fd05adb4bb11ae16/getsentry/conf/settings/prod.py#L624-L701

[thorntonryan]

Thank you for the reply!

I’ve succeeded in setting up Symbolicator as a symbol server, since i wanted to serve symbolicator from a different subdomain than sentry’s one i’ve done the following:

• Configured docker-compose.yml to bind port 81:3021 for the nginx service
• Exposed the sentry nginx on the port 81 and replicated in the new vserver the same configuration used for web and relay pointing the upstream to http://symbolicator 1
• Added the various sources to symbolicator’s config.yml

That worked perfectly fine.

Should point although that this procedure took quite a long time to figure out, documentations about sentry & symbolicator do not really explain this properly.

Thank you for the help and for bring those nice toolsets!

Originally posted by aesir on March 2021 ^1

I'm guessing this is the part I'm missing ... I'm just not quite sure what exactly the bullet points mean.

I'm pretty new to Docker, so it's probably something pretty easy/obvious for folks already familiar

[thorntonryan]

A few more observations:

1. I can reach the symbol from the sentry-self-hosted-nginx console

   # curl  http://symbolicator:3021/proxy/combase.pdb/A64E03C912C9F6071262BE39244DBAE41/combase.pdb --output -

2. I can't reach from outside docker

   curl -IL http://127.0.0.1:SENTRY_BIND/proxy/combase.pdb/A64E03C912C9F6071262BE39244DBAE41/combase.pdb

   The logs show it being redirected to auth:

   2024-05-11 09:46:30 172.20.0.1 - - [11/May/2024:14:46:30 +0000] "HEAD /symbols/proxy/combase.pdb/A64E03C912C9F6071262BE39244DBAE41/combase.pdb HTTP/1.1" 301 0 "-" "curl/7.81.0" "-"
   2024-05-11 09:46:30 172.20.0.1 - - [11/May/2024:14:46:30 +0000] "HEAD /symbols/proxy/combase.pdb/A64E03C912C9F6071262BE39244DBAE41/combase.pdb/ HTTP/1.1" 302 0 "-" "curl/7.81.0" "-"
   2024-05-11 09:46:30 172.20.0.1 - - [11/May/2024:14:46:30 +0000] "HEAD /auth/login/ HTTP/1.1" 302 0 "-" "curl/7.81.0" "-"
   2024-05-11 09:46:30 172.20.0.1 - - [11/May/2024:14:46:30 +0000] "HEAD /auth/login/sentry/ HTTP/1.1" 200 0 "-" "curl/7.81.0" 
   "-"

   And that's pretty consistent with my original / observed behavior that I was being served up HTML instead of a symbol

---

I'm attempting to go from http://127.0.0.1:SENTRY_BIND/symbols/proxy/* to http://symbolicator:3021/proxy/* Dropping the /symbols/ big and passing along everything else

I think this nginx.conf is probably pretty close, but it's clearly it's not enough:

        upstream relay {
                server relay:3000;
                keepalive 2;
        }

        upstream sentry {
                server web:9000;
                keepalive 2;
        }

+        upstream symbolicator {
+                server symbolicator:3021;
+                keepalive 2;
+        }

        server {
                listen 80;

+                location /symbols/ {
+                        proxy_pass http://symbolicator;
+                }

                location /api/store/ {
                        proxy_pass http://relay;
                }
                location ~ ^/api/[1-9]\d*/ {
                        proxy_pass http://relay;
                }
                location / {
                        proxy_pass http://sentry;
                }
                location /_static/ {
                        proxy_pass http://sentry;
                        proxy_hide_header Content-Disposition;
                }
        }

[thorntonryan]

OK, well, one approach is to sidestep self-hosted entirely.

If I do:

diff --git a/docker-compose.yml b/docker-compose.yml
index 1c6c2ad..38080c4 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -317,6 +317,8 @@ services:
   symbolicator:
     <<: *restart_policy
     image: "$SYMBOLICATOR_IMAGE"
+    ports:
+      - "3021:3021/tcp"

And expose the symbolicator container directly , then I can add the following to Visual Studio:

http://127.0.0.1:3021/proxy

Then I get loaded symbols! Woo!

[thorntonryan]

So, I think my issue was with nginx / self-hosted's proxy configuration more than it was with symbolicator.

Do you know of any way to configure self-hosted's nginx to handle forwarding to the symbolicator backend?

I guess I was hoping/assuming something like would have worked out of the box (clearly not):

http://127.0.0.1:SENTRY_BIND/symbols

Also, since I've clearly demonstrated how I can use symbolicator as a symbol server, this is clearly more a self-hosted problem. If this is the wrong place for this question, feel free to close. Or maybe it could be moved?

[thorntonryan]

Ok, another wrinkle, and this maybe more symbolicator related.

I can pull down files from the Microsoft Symbol Server redirect, but I cannot pull down individual files uploaded to my sentry project.

For example, Visual Studio is trying to query for something like:

SYMSRV:  HTTPGET: /proxy/Qt5Guid.pdb/5098B28F5F4C434696383DFC608BFB261/Qt5Guid.pdb
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND

For a symbol that lives at:

http://127.0.0.1:9887/api/0/projects/<org>/<project>/files/dsyms/?id=3

And was uploaded with:

$ ./sentry-cli.exe --auth-token <token> --url http://127.0.0.1:9887 debug-files upload Qt5Guid.pdb --project foo --log-level debug

It's looking like the symbolicator logs are reporting:

2024-05-11 10:42:37 2024-05-11T15:42:37.823535759Z ERROR symbolicator_service::download: Failed to fetch file list error=download failed: Connection refused (os error 111)

[thorntonryan]

I enabled trace logging:

# See: https://getsentry.github.io/symbolicator/#configuration
cache_dir: "/data"
bind: "0.0.0.0:3021"
logging:
-  level: "warn"
+  level: "trace"
+  format: "pretty"

And that gave better logs:

2024-05-11 21:18:12   2024-05-12T02:18:12.42824183Z DEBUG symbolicator_service::download::sentry: Fetching list of Sentry debug files from http://web/api/0/projects/sentry/foo/files/dsyms/?debug_id=<some_id>
2024-05-11 21:18:12     at crates/symbolicator-service/src/download/sentry.rs:193
2024-05-11 21:18:12 
2024-05-11 21:18:12   2024-05-12T02:18:12.48592405Z ERROR symbolicator_service::download: Failed to fetch file list, error: download failed: Connection refused (os error 111)
2024-05-11 21:18:12     at crates/symbolicator-service/src/download/mod.rs:422

So I tried to see if I could reach that endpoint from the Docer exec prompt: # curl http://web/api/0/projects/sentry/foo/files/dsyms/?debug_id=<some_id> and that failed

But adding back the port 9000 made it succeed.

And confirmed by making the change in my config:

# Configure some default symbol sources:
sources:
  # More public symbol servers be found here:
  # https://github.com/getsentry/sentry/blob/06265eed51c7f884b571bf55fe05312591ea146b/src/sentry/conf/server.py#L1894
  - id: sentry:microsoft
    type: http
    url: https://msdl.microsoft.com/download/symbols/
    layout: { type: "symstore" }
    filters: { filetypes: ["pe", "pdb", "portablepdb"] }
    is_public: true

    # In order to download debug files on demand directly from upstream sentry:
  - id: sentry:project
    type: sentry
    # Put the correct organization and project slug in the URL:
-    url: http://web/api/0/projects/sentry/foo/files/dsyms/
+    url: http://web:9000/api/0/projects/sentry/foo/files/dsyms/
    # You can use a personal auth token here that needs to have at least `project:read` permissions.
    # Create a new token here: https://sentry.io/settings/account/api/auth-tokens/
    token: <foo_token>

  # Sentry employees can find other non-public symbol servers here:
  # https://github.com/getsentry/getsentry/blob/54c52e5d844be11f7dda9c76fd05adb4bb11ae16/getsentry/conf/settings/prod.py#L624-L701

[thorntonryan]

So putting it all together, I think we have:

working.config.yml

# See: https://getsentry.github.io/symbolicator/#configuration
cache_dir: "/data"
bind: "0.0.0.0:3021"
logging:
  level: "trace"
  format: "pretty"
metrics:
  statsd: null
sentry_dsn: null # TODO: Automatically fill this with the internal project DSN

symstore_proxy: true

# Configure some default symbol sources:
sources:
  # More public symbol servers be found here:
  # https://github.com/getsentry/sentry/blob/06265eed51c7f884b571bf55fe05312591ea146b/src/sentry/conf/server.py#L1894
  - id: sentry:microsoft
    type: http
    url: https://msdl.microsoft.com/download/symbols/
    layout: { type: "symstore" }
    filters: { filetypes: ["pe", "pdb", "portablepdb"] }
    is_public: true

    # In order to download debug files on demand directly from upstream sentry:
  - id: sentry:project
    type: sentry
    # Put the correct organization and project slug in the URL:
    url: http://web:9000/api/0/projects/<org>/<project>/files/dsyms/
    # You can use a personal auth token here that needs to have at least `project:read` permissions.
    # Create a new token here: https://sentry.io/settings/account/api/auth-tokens/
    token: <token>

docker-compose.yml

diff --git a/docker-compose.yml b/docker-compose.yml
index 1c6c2ad..38080c4 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -317,6 +317,8 @@ services:
   symbolicator:
     <<: *restart_policy
     image: "$SYMBOLICATOR_IMAGE"
+    ports:
+      - "3021:3021/tcp"

Visual Studio Symbol Server Settings

• http://127.0.0.1:SENTRY_BIND/proxy

_Where SENTRYBIND is whatever port you're exposing/binding to sentry in the .env file

[loewenheim]

Hi, it looks like you already figured out your problem in the meantime, is there anything further I can help you with?

[thorntonryan]

I think we're good to close this issue. The core of it is certainly resolved.

I might try my luck opening a PR with self-hosted with the changes outlined above in https://github.com/getsentry/symbolicator/issues/1463#issuecomment-2106097532 commented out just to make it 100% obvious. It's pretty simple in hindsight, but it certainly took me ( a docker noobie) a day or two to get there

Thanks again! I look forward to playing around with the crashdump ingestion / SDK integration to see what sorts of results we can get.

[thorntonryan]

The only bits I never figured out was how to get at the symbol server behind the self-hosted nginx. If I follow, I pretty much just routed around it entirely.

If there's an obvious tip I'm missing for combining the two (and bypassing auth login), that'd be helpful, but not necessary.