"Too much data for declared Content-Length" error when activating caching

[jaepetto]

The same exact code is raising exceptions when using the @cache decorator:

from fastapi import FastAPI, Request, Response
from fastapi_redis_cache import FastApiRedisCache, cache

import base_server_info
import prometheus
import settings

logger = logging.getLogger(__name__)

app = FastAPI(title="Base Server Info", version="0.1.0")

@app.on_event("startup")
def startup_event():
    logger.info("starting up")
    redis_cache = FastApiRedisCache()
    redis_cache.init(
        host_url=f"redis://{settings.REDIS_HOST}:{settings.REDIS_PORT}",
        prefix="base-server-info",
        response_header="X-base-server-info-Cache",
        ignore_arg_types=[Request, Response],
    )

@app.get("/servers")
@cache(expire=60)
async def get_servers():
    logger.info("getting servers")

    servers = base_server_info.get_base_servers_list()
    k8sNodes = base_server_info.get_kubernetes_nodes()

    all = base_server_info.consolidate_servers_info(servers, k8sNodes)

    return_value = prometheus.prepare_for_prometheus(all)

    return return_value

This code raises the exception: "Too much data for declared Content-Length"

The same code runs well when I remove the @cache decorator (no surprise here), but also works well when the redis server is unavailable (forcing the caching mechanism to evaluate the code).

Thanks in advance for any insight...

The stack trace is as below:

poc_prometheus_http_service_discovery-app-1    | WARNING:  StatReload detected changes in 'main.py'. Reloading...
poc_prometheus_http_service_discovery-app-1    | INFO:     Shutting down
poc_prometheus_http_service_discovery-app-1    | INFO:     Waiting for application shutdown.
poc_prometheus_http_service_discovery-app-1    | INFO:     Application shutdown complete.
poc_prometheus_http_service_discovery-app-1    | INFO:     Finished server process [8]
poc_prometheus_http_service_discovery-app-1    | INFO:     Started server process [9]
poc_prometheus_http_service_discovery-app-1    | INFO:     Waiting for application startup.
poc_prometheus_http_service_discovery-app-1    | INFO:fastapi_redis_cache.client: 10/25/2022 10:53:57 AM | CONNECT_BEGIN: Attempting to connect to Redis server...
poc_prometheus_http_service_discovery-app-1    | INFO:fastapi_redis_cache.client: 10/25/2022 10:53:57 AM | CONNECT_SUCCESS: Redis client is connected to server.
poc_prometheus_http_service_discovery-app-1    | INFO:     Application startup complete.
poc_prometheus_http_service_discovery-app-1    | INFO:     172.19.0.1:42710 - "GET /servers HTTP/1.1" 200 OK
poc_prometheus_http_service_discovery-app-1    | WARNING:  StatReload detected changes in 'main.py'. Reloading...
poc_prometheus_http_service_discovery-app-1    | INFO:     Shutting down
poc_prometheus_http_service_discovery-app-1    | INFO:     Waiting for application shutdown.
poc_prometheus_http_service_discovery-app-1    | INFO:     Application shutdown complete.
poc_prometheus_http_service_discovery-app-1    | INFO:     Finished server process [9]
poc_prometheus_http_service_discovery-app-1    | INFO:     Started server process [19]
poc_prometheus_http_service_discovery-app-1    | INFO:     Waiting for application startup.
poc_prometheus_http_service_discovery-app-1    | INFO:fastapi_redis_cache.client: 10/25/2022 10:57:05 AM | CONNECT_BEGIN: Attempting to connect to Redis server...
poc_prometheus_http_service_discovery-app-1    | INFO:fastapi_redis_cache.client: 10/25/2022 10:57:05 AM | CONNECT_SUCCESS: Redis client is connected to server.
poc_prometheus_http_service_discovery-app-1    | INFO:     Application startup complete.
poc_prometheus_http_service_discovery-app-1    | INFO:fastapi_redis_cache.client: 10/25/2022 10:57:11 AM | KEY_ADDED_TO_CACHE: key=base-server-info:main.get_servers()
poc_prometheus_http_service_discovery-app-1    | INFO:     172.19.0.1:54082 - "GET /servers HTTP/1.1" 200 OK
poc_prometheus_http_service_discovery-app-1    | ERROR:    Exception in ASGI application
poc_prometheus_http_service_discovery-app-1    | Traceback (most recent call last):
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
poc_prometheus_http_service_discovery-app-1    |     result = await app(  # type: ignore[func-returns-value]
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
poc_prometheus_http_service_discovery-app-1    |     return await self.app(scope, receive, send)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 270, in __call__
poc_prometheus_http_service_discovery-app-1    |     await super().__call__(scope, receive, send)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
poc_prometheus_http_service_discovery-app-1    |     await self.middleware_stack(scope, receive, send)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
poc_prometheus_http_service_discovery-app-1    |     raise exc
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
poc_prometheus_http_service_discovery-app-1    |     await self.app(scope, receive, _send)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 75, in __call__
poc_prometheus_http_service_discovery-app-1    |     raise exc
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 64, in __call__
poc_prometheus_http_service_discovery-app-1    |     await self.app(scope, receive, sender)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
poc_prometheus_http_service_discovery-app-1    |     raise e
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
poc_prometheus_http_service_discovery-app-1    |     await self.app(scope, receive, send)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 680, in __call__
poc_prometheus_http_service_discovery-app-1    |     await route.handle(scope, receive, send)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 275, in handle
poc_prometheus_http_service_discovery-app-1    |     await self.app(scope, receive, send)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 68, in app
poc_prometheus_http_service_discovery-app-1    |     await response(scope, receive, send)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 167, in __call__
poc_prometheus_http_service_discovery-app-1    |     await send({"type": "http.response.body", "body": self.body})
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 61, in sender
poc_prometheus_http_service_discovery-app-1    |     await send(message)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 159, in _send
poc_prometheus_http_service_discovery-app-1    |     await send(message)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 512, in send
poc_prometheus_http_service_discovery-app-1    |     output = self.conn.send(event)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/h11/_connection.py", line 512, in send
poc_prometheus_http_service_discovery-app-1    |     data_list = self.send_with_data_passthrough(event)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/h11/_connection.py", line 545, in send_with_data_passthrough
poc_prometheus_http_service_discovery-app-1    |     writer(event, data_list.append)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/h11/_writers.py", line 65, in __call__
poc_prometheus_http_service_discovery-app-1    |     self.send_data(event.data, write)
poc_prometheus_http_service_discovery-app-1    |   File "/usr/local/lib/python3.10/site-packages/h11/_writers.py", line 91, in send_data
poc_prometheus_http_service_discovery-app-1    |     raise LocalProtocolError("Too much data for declared Content-Length")
poc_prometheus_http_service_discovery-app-1    | h11._util.LocalProtocolError: Too much data for declared Content-Length

[jaeger-2601]

+1

[jaepetto]

@jaeger-2601 I made a PR for that. In the meantime, you can apply the below patch:

--- fastapi_redis_cache_cache_old.py    2022-10-25 16:24:43.000000000 +0200
+++ fastapi_redis_cache_cache_new.py    2022-10-25 16:24:07.000000000 +0200
@@ -34,6 +34,8 @@
             create_response_directly = not response
             if create_response_directly:
                 response = Response()
+                if "content-length" in response.headers.keys():
+                    del response.headers["content-length"]
             redis_cache = FastApiRedisCache()
             if redis_cache.not_connected or redis_cache.request_is_not_cacheable(request):
                 # if the redis client is not connected or request is not cacheable, no caching behavior is performed.

[LittleRookie1706]

+1

[jaeger-2601]

@LittleRookie1706 This project seems to be abandoned by the maintainer. I suggest moving to fastapi-cache or you can apply @jaepetto's patch.

[jaepetto]

Thanks @jaeger-2601 for the tip about fastapi-cache. I'll probably move to it too.