search

trallnag / prometheus-fastapi-instrumentator

Instrument your FastAPI with Prometheus metrics.
ISC License
948 stars 84 forks source link

the middleware can't be installed for the actual FastAPI version #214

Closed alexted closed 1 year ago

alexted commented 1 year ago

Hi! Since FastAPI version 0.91.0, I started getting the following error when trying to run the application:

INFO:     Will watch for changes in these directories: ['/home/my_name/PycharmProjects/my-app']
INFO:     Uvicorn running on http://127.0.0.1:5000 (Press CTRL+C to quit)
INFO:     Started reloader process [15443] using WatchFiles
INFO:     Started server process [15472]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 648, in startup
    await handler()
  File "/home/my_name/PycharmProjects/my-app/core/app/service.py", line 75, in integrate_monitoring_plugin
    Instrumentator(excluded_handlers=["/health", "/metrics"]).instrument(app).expose(app, include_in_schema=False)
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/instrumentation.py", line 121, in instrument
    app.add_middleware(
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/applications.py", line 135, in add_middleware
    raise RuntimeError("Cannot add middleware after an application has started")
RuntimeError: Cannot add middleware after an application has started

ERROR:    Application startup failed. Exiting.

I initiate the application this way:

def create_app() -> FastAPI:
    app = FastAPI()

    #blah-blah-blah

    @app.on_event("startup")
    async def integrate_monitoring_plugin():
        Instrumentator(excluded_handlers=["/health", "/metrics"]).instrument(app).expose(app, include_in_schema=False)

    return app
Dencrash commented 1 year ago

We have the same problem with both the fastapi versions 0.91.0 and 0.92.0 being effected. It seems like adding a middleware after startup is not valid anymore, though they also fixed the fact that middlewares are initialized multiple times. Is there an alternative way to integrate the instrumentator?

duffn commented 1 year ago

We're encountering the same issue. Here's where this was introduced in Starlette. https://github.com/encode/starlette/commit/51c1de1839ee187bb20de5748fca23baa37369c6#diff-3eb07c0eea72d98f1b844e07844c2a68b7c11c63956b67d5afdd9c7f79ab946fR135

harochau commented 1 year ago

I have the same problem. Maybe we can use https://github.com/stephenhillier/starlette_exporter instead? Does anybody have experience with that library, is it a drop-in replacement?

alexted commented 1 year ago

Yes, I too am thinking of switching to another library.

alfaro28 commented 1 year ago

Is there any reason to implement the instrumentator on the "startup" event? Just moving the instrumentator outside of the event makes things work

alexted commented 1 year ago

This works until you run unit tests.

jonathanloske commented 1 year ago

Is there any reason to implement the instrumentator on the "startup" event? Just moving the instrumentator outside of the event makes things work

According to the maintainer:

Prometheus expects a fully initialized ASGI client, where uvicorn now delays execution in newer versions. Closing.

alfaro28 commented 1 year ago

sorry I should have been more clear, what I'm currently doing is creating the instrumentator and "instrument" the app outside of the startup event, but actually "exposing" it on the startup event and seems to be working fine

duffn commented 1 year ago

sorry I should have been more clear, what I'm currently doing is creating the instrumentator and "instrument" the app outside of the startup event, but actually "exposing" it on the startup event and seems to be working fine

Can you share a bit of code here please?

alfaro28 commented 1 year ago

Sure,, just a quick note, I'm using start_http_server because I run my prometheus server on a diferent port, but I asume you could save the Instrumentator variable and just call .expose on the startup_event function

from fastapi import FastAPI
from prometheus_client import start_http_server
from prometheus_fastapi_instrumentator import Instrumentator

async def startup_event():
    start_http_server(8080)

app = FastAPI(
    on_startup=[startup_event]
)

Instrumentator(
    excluded_handlers=["/metrics"],
).instrument(app)
harochau commented 1 year ago

can confirm, this works with the new Fastapi

instrumentator = Instrumentator().instrument(app)
@app.on_event("startup")
async def _startup():
    instrumentator.expose(app, endpoint='/metrics', tags=['metrics'])
trallnag commented 1 year ago

Is there any reason to implement the instrumentator on the "startup" event? Just moving the instrumentator outside of the event makes things work

The README was updated to use startup around 6 months ago due to #80. It is probably best to update the README again with the next release. Just tried it (while testing other things) and for me it works without startup fine.

Here are relevant dependencies:

❯ pip list
Package                           Version Editable project location
--------------------------------- ------- ----------------------------
fastapi                           0.92.0
prometheus-client                 0.16.0
prometheus-fastapi-instrumentator 100.0.0
starlette                         0.25.0
uvicorn                           0.20.0

The code:

from fastapi import FastAPI

from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI()

@app.get("/app")
def read_main():
    return {"message": "Hello World from main app"}

Instrumentator().instrument(app).expose(app)

Reproducible examples would be great as I am not using the package actively anymore.

alexted commented 1 year ago

it breaks all my unit tests: image

trallnag commented 1 year ago

@alexted does this also happen if you use https://github.com/stephenhillier/starlette_exporter? I think it is related to how the prometheus client library works. In the unit tests of this package here I call the following function at the beginning of every relevant unit test: https://github.com/trallnag/prometheus-fastapi-instrumentator/blob/master/tests/helpers/utils.py#L4

Another solution seems to be to use a new registry for every unit test: https://github.com/rycus86/prometheus_flask_exporter/issues/17

Current version of prometheus-fastapi-instrumentator does not support passing custom registry, but this feature has already been merged as part of clean up and I will release it today

saschnet commented 1 year ago

Is there anything pointing against the solution by @alfaro28 / @harochau?

For me, it is working with all circumstances (unit tests + the app itself) and is much simpler than adding a new registry for every unit test etc.

trallnag commented 1 year ago

@saschnet, I don't know 😆 I will update the README and switch to the example provided by @harochau

alexted commented 1 year ago

@alexted does this also happen if you use https://github.com/stephenhillier/starlette_exporter?

No, everything works fine when using this library. The application runs and the unit tests pass.

alexted commented 1 year ago

@trallnag thanks for the release. with the new version I started to get another error when running unit tests:


tests/v1_1/advertisers/test_get_expenses.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1757: in get
    return await self.request(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1533: in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1620: in send
    response = await self._send_handling_auth(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1648: in _send_handling_auth
    response = await self._send_handling_redirects(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1685: in _send_handling_redirects
    response = await self._send_single_request(request)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1722: in _send_single_request
    response = await transport.handle_async_request(request)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_transports/asgi.py:162: in handle_async_request
    await self.app(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/fastapi/applications.py:271: in __call__
    await super().__call__(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/applications.py:118: in __call__
    await self.middleware_stack(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py:184: in __call__
    raise exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py:162: in __call__
    await self.app(scope, receive, _send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py:84: in __call__
    await self.app(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:108: in __call__
    response = await self.dispatch_func(request, call_next)
src/utils/middlewares/log_requests.py:19: in log_requests
    response_body = [chunk async for chunk in response.body_iterator]
src/utils/middlewares/log_requests.py:19: in <listcomp>
    response_body = [chunk async for chunk in response.body_iterator]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:109: in __call__
    await response(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:270: in __call__
    async with anyio.create_task_group() as task_group:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:662: in __aexit__
    raise exceptions[0]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:273: in wrap
    await func()
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:134: in stream_response
    return await super().stream_response(send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:262: in stream_response
    async for chunk in self.body_iterator:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:109: in __call__
    await response(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:270: in __call__
    async with anyio.create_task_group() as task_group:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:662: in __aexit__
    raise exceptions[0]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:273: in wrap
    await func()
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:134: in stream_response
    return await super().stream_response(send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:262: in stream_response
    async for chunk in self.body_iterator:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/middleware.py:181: in __call__
    instrumentation(info)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

info = <prometheus_fastapi_instrumentator.metrics.Info object at 0x7feb23667fd0>

    def instrumentation(info: Info) -> None:
>       TOTAL.labels(info.method, info.modified_status, info.modified_handler).inc()
E       NameError: free variable 'TOTAL' referenced before assignment in enclosing scope

../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/metrics.py:625: NameError
AliSayyah commented 1 year ago

For now, I'm excluding 'prometheus_fastapi_instrumentator' when running tests. I'm using 'pytest' so incase anyone having the same problem:

instrumentor = None
    if "pytest" not in sys.modules:
        instrumentor = PrometheusFastApiInstrumentator(
            should_group_status_codes=False
        ).instrument(
            app,
        )

and:

    @app.on_event("startup")
    async def _startup() -> None: 
        ...
        if "pytest" not in sys.modules:
            setup_prometheus(app, instrumentor)
        ...

note: this would only work if you haven't imported 'pytest' outside your tests. there may be cleaner ways to check this.

trallnag commented 1 year ago

@alexted, probably related to #219

trallnag commented 1 year ago

The NameError thing should be fixed now with the latest release https://github.com/trallnag/prometheus-fastapi-instrumentator/releases/tag/v5.11.0

AliSayyah commented 1 year ago

The NameError thing should be fixed now with the latest release https://github.com/trallnag/prometheus-fastapi-instrumentator/releases/tag/v5.11.0

Thanks. NameError is gone.

alexted commented 1 year ago

ty @trallnag