Missing source info: Project not found, sentry-cli reports HTTP 302

[vmandic]

Package

Sentry.AspNetCore

.NET Flavor

.NET

.NET Version

7.0.203

OS

Linux

SDK Version

3.31.0

Self-Hosted Sentry Version

No response

Steps to Reproduce

We observed a strange issue after upgrading from netcoreapp3.1 to net7 and disabling PublishSingleFile (due to #2362 which we also reported) (and thus also disabling ReadyToRun) and opting into TieredPGO=true; now debug/source upload is failing in our build pipeline job during dotnet publish and we lack source code in Sentry UI, and the error tells us that SentryCLI can't find the project like /source/Company.Project/Company.Project.csproj ... and then of course I went to check the docker image state, ie. the dirs and files at that point of dotnet publish (and before it) and the .csproj files ARE there.

Here is the upload log output part from our docker build:

#29 [build 21/23] RUN dotnet publish -c release -o /app -r linux-musl-x64 --...
#29 1.365 MSBuild version 17.5.1+f6fdcf537 for .NET
#29 19.64   Company.Project1 -> /source/Company.Project1/bin/linux-musl-x64/release/net7.0/Company.Project1.dll
#29 19.70     INFO    2023-05-31 10:41:39.575846132 +00:00 sentry-cli was invoked with the following command line: "/root/.nuget/packages/sentry/3.31.0/tools/sentry-cli-Linux-x86_64" "info" "--no-defaults"
#29 20.25   Preparing to upload debug symbols to Sentry for Company.Project1 (release/net7.0)
#29 20.26     INFO    2023-05-31 10:41:40.147081323 +00:00 sentry-cli was invoked with the following command line: "/root/.nuget/packages/sentry/3.31.0/tools/sentry-cli-Linux-x86_64" "debug-files" "upload" "--org" "Company" "--project" "Company-api" "bin/linux-musl-x64/release/net7.0/"
#29 20.26     INFO    2023-05-31 10:41:40.147213440 +00:00 Issuing a command for Organization: Company Project: Company-api
#29 20.81   > Found 1 debug information file
#29 20.81   > Prepared debug information file for upload
#29 21.04 EXEC : warning : project not found [/source/Company.Project1/Company.Project1.csproj]
#29 21.04 /root/.nuget/packages/sentry/3.31.0/buildTransitive/Sentry.targets(132,5): warning : Sentry CLI could not upload debug files. [/source/Company.Project1/Company.Project1.csproj]
#29 21.04   Preparing to upload sources to Sentry for Company.Project1 (release/net7.0)
#29 21.04     INFO    2023-05-31 10:41:40.923452001 +00:00 sentry-cli was invoked with the following command line: "/root/.nuget/packages/sentry/3.31.0/tools/sentry-cli-Linux-x86_64" "debug-files" "bundle-sources" "obj/linux-musl-x64/release/net7.0/Company.Project1.pdb"
#29 21.05   /source/Company.Project1/obj/linux-musl-x64/release/net7.0/Company.Project1.src.zip
#29 21.06     INFO    2023-05-31 10:41:40.946147695 +00:00 sentry-cli was invoked with the following command line: "/root/.nuget/packages/sentry/3.31.0/tools/sentry-cli-Linux-x86_64" "debug-files" "upload" "--org" "Company" "--project" "Company-api" "obj/linux-musl-x64/release/net7.0/Company.Project1.src.zip"
#29 21.06     INFO    2023-05-31 10:41:40.946233570 +00:00 Issuing a command for Organization: Company Project: Company-api
#29 21.61   > Found 1 debug information file (1 with embedded sources)
#29 21.61   > Prepared debug information file for upload
#29 21.76 EXEC : warning : project not found [/source/Company.Project1/Company.Project1.csproj]
#29 21.77 /root/.nuget/packages/sentry/3.31.0/buildTransitive/Sentry.targets(154,5): warning : Sentry CLI could not upload sources. [/source/Company.Project1/Company.Project1.csproj]
#29 32.63   Company.Project2 -> /source/Company.Project2/bin/linux-musl-x64/release/net7.0/Company.Project2.dll
#29 32.74     INFO    2023-05-31 10:41:52.604116763 +00:00 sentry-cli was invoked with the following command line: "/root/.nuget/packages/sentry/3.31.0/tools/sentry-cli-Linux-x86_64" "info" "--no-defaults"
#29 33.29   Preparing to upload debug symbols to Sentry for Company.Project2 (release/net7.0)

... and so on for 5 more projects including exe one at the end

And if you wonder what the core parts of dockerfile look, here it is:

FROM mcr.microsoft.com/dotnet/sdk:7.0.203-alpine3.17 AS build
WORKDIR /source

... # some other files copied

# copy csproj and restore as distinct layers
COPY Company.Project0/*.csproj Company.Project0/
COPY Company.Project1/*.csproj Company.Project1/
COPY Company.Project2/*.csproj Company.Project2/
COPY Company.Project3/*.csproj Company.Project3/
COPY Company.Project4/*.csproj Company.Project4/
COPY $PROJECT/*.csproj $PROJECT/
ENV DOTNET_ReadyToRun=0
RUN dotnet restore -r $PLATFORM $PROJECT/$PROJECT.csproj

# copy everything else and build app
COPY Company.Project0/ Company.Project0/
COPY Company.Project1/ Company.Project1/
COPY Company.Project2/ Company.Project2/
COPY Company.Project3/ Company.Project3/
COPY Company.Project4/ Company.Project4/
COPY $PROJECT/ $PROJECT/

WORKDIR /source/$PROJECT
# Ensure that appsettings.json is in the new workdir so publish does not fail, might be a Windows only thing
COPY appsettings.json .

# must be set before dotnet publish to be able to upload source and debug data related to the release
ARG SENTRY_CMDS="-p:UseSentryCLI=true -p:SentryOrg=Company \
-p:SentryProject=$SENTRY_PROJECT_NAME -p:SentryUploadSymbols=true -p:SentryUploadSources=true"
ENV SENTRY_LOG_LEVEL=info

... # some envs

# Builds the dotnet project, prepares debug and source info, uploads it to Sentry
# Sentry CLI requires SENTRY_AUTH_TOKEN in env
RUN dotnet publish -c release -o /app -r $PLATFORM --no-restore -p:TieredPGO=true \
    --self-contained true -p:DebugType=Portable -p:DebugSymbols=true \
    -p:Version=$SEMVER_BUILD_DT_UTC-sha-$COMMIT_SHA_SHORT $SENTRY_CMDS

... # app image switch to mcr.microsoft.com/dotnet/runtime-deps:7.0-alpine, install some apks and side-car stuff

ARG PROJECT
ENV PROJECT=$PROJECT
ENTRYPOINT exec ./$PROJECT

So I built the container locally and really verified the files are there, before and after publish.

Here is a screenshot from the container:

I enabled the debug logs (locally testing this and also same issue as on our CI job) for Sentry CLi and here is more detailed output for first project:

#25 27.49     DEBUG   2023-06-01 08:11:49.529978635 +00:00 body: {"url":"https://sentry.io/api/0/organizations/Company/chunk-upload/","chunkSize":8388608,"chunksPerRequest":64,"maxFileSize":2147483648,"maxRequestSize":33554432,"concurrency":8,"hashAlgorithm":"sha1","compression":["gzip"],"accept":["debug_files","release_files","pdbs","sources","bcsymbolmaps","il2cpp","portablepdbs","artifact_bundles"]}
#25 27.56   > Found 1 debug information file
#25 27.56   > Prepared debug information file for upload
#25 27.56     DEBUG   2023-06-01 08:11:49.608519677 +00:00 request POST https://sentry.io/api/0/projects/Company/Company-api/files/difs/assemble/
#25 27.56     DEBUG   2023-06-01 08:11:49.608549969 +00:00 using token authentication
#25 27.57     DEBUG   2023-06-01 08:11:49.615829510 +00:00 json body: {"ecf506e4c38f17d9c510db6925014373667139c5":{"name":"Company.Project1.pdb","debug_id":"d2bf96a9-acb6-4ca8-a10e-d3dfb26fcf5f-b20b52c1","chunks":["ecf506e4c38f17d9c510db6925014373667139c5"]}}
#25 27.57     DEBUG   2023-06-01 08:11:49.615921219 +00:00 retry number 0, max retries: 5
#25 27.57     DEBUG   2023-06-01 08:11:49.617567760 +00:00 > POST /api/0/projects/Company/Company-api/files/difs/assemble/ HTTP/1.1
#25 27.57     DEBUG   2023-06-01 08:11:49.617636760 +00:00 > Host: sentry.io
#25 27.57     DEBUG   2023-06-01 08:11:49.617642094 +00:00 > Accept: */*
#25 27.57     DEBUG   2023-06-01 08:11:49.617647677 +00:00 > Connection: TE
#25 27.57     DEBUG   2023-06-01 08:11:49.617651844 +00:00 > TE: gzip
#25 27.57     DEBUG   2023-06-01 08:11:49.617656052 +00:00 > User-Agent: sentry-cli/2.17.5
#25 27.57     DEBUG   2023-06-01 08:11:49.619191552 +00:00 > Authorization: Bearer 2253eb57***
#25 27.57     DEBUG   2023-06-01 08:11:49.619208427 +00:00 > Content-Type: application/json
#25 27.57     DEBUG   2023-06-01 08:11:49.619213094 +00:00 > Content-Length: 196
#25 27.79     DEBUG   2023-06-01 08:11:49.838609844 +00:00 < HTTP/1.1 302 Found
#25 27.79     DEBUG   2023-06-01 08:11:49.838760927 +00:00 < Server: nginx
#25 27.79     DEBUG   2023-06-01 08:11:49.838839594 +00:00 < Date: Thu, 01 Jun 2023 08:11:49 GMT
#25 27.79     DEBUG   2023-06-01 08:11:49.839006510 +00:00 < Content-Type: application/json
#25 27.79     DEBUG   2023-06-01 08:11:49.839020260 +00:00 < Content-Length: 165
#25 27.79     DEBUG   2023-06-01 08:11:49.839028969 +00:00 < Connection: keep-alive
#25 27.79     DEBUG   2023-06-01 08:11:49.839037469 +00:00 < location: /api/0/projects/Company/api/files/difs/assemble/
#25 27.80     DEBUG   2023-06-01 08:11:49.839044635 +00:00 < allow: POST, OPTIONS
#25 27.80     DEBUG   2023-06-01 08:11:49.839053510 +00:00 < access-control-allow-methods: POST, OPTIONS
#25 27.80     DEBUG   2023-06-01 08:11:49.839269219 +00:00 < access-control-allow-headers: X-Sentry-Auth, X-Requested-With, Origin, Accept, Content-Type, Authentication, Authorization, Content-Encoding, sentry-trace, baggage, X-CSRFToken
#25 27.81     DEBUG   2023-06-01 08:11:49.839299969 +00:00 < access-control-expose-headers: X-Sentry-Error, Retry-After
#25 27.81     DEBUG   2023-06-01 08:11:49.839308552 +00:00 < access-control-allow-origin: *
#25 27.81     DEBUG   2023-06-01 08:11:49.839314344 +00:00 < x-sentry-rate-limit-remaining: 39
#25 27.81     DEBUG   2023-06-01 08:11:49.839320677 +00:00 < x-sentry-rate-limit-limit: 40
#25 27.81     DEBUG   2023-06-01 08:11:49.839326552 +00:00 < x-sentry-rate-limit-reset: 1685607110
#25 27.81     DEBUG   2023-06-01 08:11:49.839332219 +00:00 < x-sentry-rate-limit-concurrentremaining: 24
#25 27.81     DEBUG   2023-06-01 08:11:49.839338302 +00:00 < x-sentry-rate-limit-concurrentlimit: 25
#25 27.81     DEBUG   2023-06-01 08:11:49.839348094 +00:00 < vary: Accept-Language, Cookie
#25 27.81     DEBUG   2023-06-01 08:11:49.839361052 +00:00 < content-language: en
#25 27.81     DEBUG   2023-06-01 08:11:49.839369052 +00:00 < x-frame-options: deny
#25 27.81     DEBUG   2023-06-01 08:11:49.839374844 +00:00 < x-content-type-options: nosniff
#25 27.81     DEBUG   2023-06-01 08:11:49.839380302 +00:00 < x-xss-protection: 1; mode=block
#25 27.81     DEBUG   2023-06-01 08:11:49.839394969 +00:00 < content-security-policy: object-src 'self'; default-src *; style-src * 'unsafe-inline'; font-src * data:; img-src * blob: data:; connect-src *; script-src 'self' 'unsafe-inline' 'report-sample' 'unsafe-eval' s1.sentry-cdn.com js.sentry-cdn.com browser.sentry-cdn.com statuspage-production.s3.amazonaws.com static.zdassets.com aui-cdn.atlassian.com connect-cdn.atl-paas.net js.stripe.com ssl.google-analytics.com 'strict-dynamic' cdn.pendo.io data.pendo.io pendo-io-static.storage.googleapis.com pendo-static-5634074999128064.storage.googleapis.com; frame-ancestors 'self'; base-uri 'none'; report-uri https://o1.ingest.sentry.io/api/54785/security/?sentry_key=[REDACTED]&sentry_release=4ae4036c3c5a30a9bb46b549c23eee81a8a6770f
#25 27.81     DEBUG   2023-06-01 08:11:49.839409677 +00:00 < x-envoy-attempt-count: 1
#25 27.81     DEBUG   2023-06-01 08:11:49.839421219 +00:00 < x-envoy-upstream-service-time: 53
#25 27.81     DEBUG   2023-06-01 08:11:49.839432677 +00:00 < x-served-by: getsentry-web-default-common-production-685596cfc6-4rgzv
#25 27.81     DEBUG   2023-06-01 08:11:49.839452844 +00:00 < x-served-by: lb-10
#25 27.81     DEBUG   2023-06-01 08:11:49.839461552 +00:00 < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
#25 27.81     DEBUG   2023-06-01 08:11:49.839525927 +00:00 response status: 302
#25 27.81     DEBUG   2023-06-01 08:11:49.839545302 +00:00 body: {"slug":"api","detail":{"code":"resource-moved","message":"Resource has been moved","extra":{"url":"/api/0/projects/Company/api/files/difs/assemble/","slug":"api"}}}
#25 27.94 EXEC : warning : project not found [/source/Company.Project1/Company.Project1.csproj]
#25 27.94     DEBUG   2023-06-01 08:11:49.845149552 +00:00 skipping update nagger because session is not attended
#25 27.94 /root/.nuget/packages/sentry/3.31.0/buildTransitive/Sentry.targets(132,5): warning : Sentry CLI could not upload debug files. [/source/Company.Project1/Company.Project1.csproj]
#25 27.94   Preparing to upload sources to Sentry for Company.Project1 (release/net7.0)
#25 28.34     DEBUG   2023-06-01 08:11:50.385500594 +00:00 sentry-cli version: 2.17.5, platform: "linux", architecture: "aarch64"
#25 28.34     INFO    2023-06-01 08:11:50.386014511 +00:00 sentry-cli was invoked with the following command line: "/root/.nuget/packages/sentry/3.31.0/tools/sentry-cli-Linux-aarch64" "debug-files" "bundle-sources" "obj/linux-musl-x64/release/net7.0/Company.Project1.pdb"

Same issue repeats for .pdb file.

Any ideas please? For now I will disable uploading sources as I can't see a way to pass this. :(

Sample issue ID where we noticed this: 4218503983, Project ID: 6323756

Expected Result

We expect the upload of sources and .pbd to pass as the .csproj files ARE in the place where the error reports they are not?!

Actual Result

Upload is skipped and the issues in Sentry lack source info and we cant view source code in the Sentry UI. Stack trace and line numbers are OK.

[vmandic]

Not sure if I should've reported this to sentry-cli project as I looked up for this specific error string in your source and found some rust code possibly related to this: https://github.com/getsentry/sentry-cli/blob/033bd2173e3aeb82941913efdee4b80add99ea0c/src/api.rs#L1302-L1311

Seems HTTP 301/302 is handled as this error and you can see I exactly get it back: #25 27.81 DEBUG 2023-06-01 08:11:49.839525927 +00:00 response status: 302?

Do you know maybe @bruno-garcia anything about this (asking you also because this might be a wrong project I reported to)?

[mattjohnsonpint]

Hi. Sorry for the delay.

When sentry-cli reports "project not found" - it means that the combined org name and project name are not found using the configured auth token.

In your dockerfile, you specify -p:SentryOrg=Company -p:SentryProject=$SENTRY_PROJECT_NAME. I presume Company is replaced with your actual Sentry organization slug. Perhaps $SENTRY_PROJECT_NAME is not resolving correctly?

Also, you can omit -p:UseSentryCLI=true unless you've disabled it elsewhere. It doesn't hurt, it's just redundant.

Personally, I prefer to set most of the MSBuild properties in the .csproj files themselves, or in a root Directory.Build.props - but the command line approach should work the same.

You might try manually checking with sentry-cli --auth-token [your-auth-token] projects list --org [your-sentry-org] to see if the project names are returned correctly. You might also want to echo the $SentryProject variable from your dockerfile to make sure it's passed properly.

You might also be running into issues with the way dockerfile stages use environment variables. If you've got a multi-stage build (separated by FROM statements in your dockerfile), then make sure the environment variable is set in the stage you're needing to use it. For example, if you're using the dockerfile standard file that's created by VS and Rider (for an ASP.NET Core project), it has build and publish in separate phases. You only need to use Sentry CLI in the build phase.

FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app
ENV ASPNETCORE_URLS=http://+:8080
EXPOSE 8080

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build

# Pass when building the container:  docker build --build-arg SENTRY_AUTH_TOKEN=[your-auth-token]
ARG SENTRY_AUTH_TOKEN
ENV SENTRY_AUTH_TOKEN=$SENTRY_AUTH_TOKEN

WORKDIR /src
COPY ["MyWebApp.csproj", "./"]
RUN dotnet restore "MyWebApp.csproj"
COPY . .
WORKDIR "/src/"
RUN dotnet build "MyWebApp.csproj" --no-restore -c Release -o /app/build

# In the publishing phase, we don't need to use Sentry CLI because it was already used in the build phase.
FROM build AS publish
RUN dotnet publish "MyWebApp.csproj" -c Release -o /app/publish -p:UseAppHost=false -p:UseSentryCLI=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "MyWebApp.dll"]

The above presumes setting <SentryOrg>, <SentryProject>, <SentryUploadSymbols>, and <SentryUploadSources> in the MyWebApp.csproj file, or in a Directory.Build.props file. Though if you prefer, you could pass them in the same way the auth token is passed.

Hope that helps.

[vmandic]

@mattjohnsonpint, kind of a shame I did not verify names first, somehow we changed the names as a part of some cleanup and OFC it did not work. All good, thank you for the effort! Thanks, V.