Multiple endpoints being profiled together

[antoineqian]

Environment

SaaS (https://sentry.io/)

What are you trying to accomplish?

I am trying to profile code relating to endpoints I defined and added to my fastapi app.

How are you getting stuck?

I setup sentry with similar parameters to the tutorial and i initialize the app with lifespan.

import sentry_sdk
from sentry_sdk.integrations.starlette import StarletteIntegration
from sentry_sdk.integrations.fastapi import FastApiIntegration

@asynccontextmanager
async def lifespan(app: FastAPI):

   sentry_sdk.init(
    enable_tracing=True,
        dsn=settings.SENTRY_DSN,
        environment=settings.ENV.value,
        traces_sample_rate=settings.SENTRY_TRACE_SAMPLE_RATE,
        profiles_sample_rate=settings.SENTRY_PROFILE_SAMPLE_RATE,
    integrations=[
        StarletteIntegration(
            transaction_style="endpoint"
        ),
        FastApiIntegration(
            transaction_style="endpoint"
        ),
    ],
)

app = FastAPI(lifespan=lifespan, docs_url="/app/docs")

app.include_router()
app.include_router()

I add multiple routers in my app which define several endpoints. I can see the endpoints in the sentry app, but when in the profiling panel, I can see that the profile of an endpoint contains profiling of code of another endpoint entirely, or of some background tasks added by another endpoint, not what one would expect at all.

For example, I can show that Router.call appears several times in the profiling of this endpoint

Can someone explains why this happens?

Where in the product are you?

Profiling

Link

No response

DSN

No response

Version

fastapi==0.95.1 sentry-sdk==1.45.0

[getsantry[bot]]

Assigning to @getsentry/support for routing ⏲️

[getsantry[bot]]

Routing to @getsentry/product-owners-profiling for triage ⏲️

[Zylphrex]

Hi @antoineqian, what you're seeing is a consequence of how asyncio works in Python and how profiling works. When using asyncio, python runs things inside of coroutines that all execute within the same thread (You can see this in your screenshot where it shows that this is the MainThread). The profiling feature profiles the program per thread. So you see the underlying execution model is exposed in the profile where it shows multiple coroutines interweaved.

I agree this is not idea but that is the limitation of this per thread profiling we have today but attributing every stack collected to a specific transaction/endpoint is nontrivial and will likely increase the overhead on the application.

[antoineqian]

Thanks for the reply. Are there any workarounds that exist? I'd think many people would use asyncio with FastApi

[Zylphrex]

I'm not aware of any workarounds that exist. The closest I've seen is the use of sync handlers in FastAPI as under the hood, Starlette will delegate these requests to a separate thread which will be profiled per transaction but that would mean preferring threads over asyncio.