Unhandled Redix error

[Huskydog9988]

It seems like this error is caused by trying to access a key that doesn't exist, at least acording to my quick digging. If its any help to you, we are using https://github.com/upstash/ratelimit.

10:42:46.423 [error] #PID<0.14839.0> running Srh.Http.BaseRouter (connection #PID<0.14837.0>, stream id 1) terminated
Server: example.example.svc.cluster.local:80 (http)
Request: POST /
** (exit) an exception was raised:
    ** (KeyError) key :message not found in: %Redix.ConnectionError{reason: :closed}
        (srh 0.1.0) lib/srh/http/command_handler.ex:160: Srh.Http.CommandHandler.dispatch_command/2
        (srh 0.1.0) lib/srh/http/base_router.ex:16: anonymous fn/2 in Srh.Http.BaseRouter.do_match/4
        (srh 0.1.0) lib/plug/router.ex:246: anonymous fn/4 in Srh.Http.BaseRouter.dispatch/2
        (telemetry 1.1.0) /app/deps/telemetry/src/telemetry.erl:320: :telemetry.span/3
        (srh 0.1.0) lib/plug/router.ex:242: Srh.Http.BaseRouter.dispatch/2
        (srh 0.1.0) lib/srh/http/base_router.ex:1: Srh.Http.BaseRouter.plug_builder_call/2
        (plug_cowboy 2.5.2) lib/plug/cowboy/handler.ex:12: Plug.Cowboy.Handler.init/2
        (cowboy 2.9.0) /app/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2

[hiett]

Hi! At a first glance from this error, the actual Redis connection is dying and is closed. I will take a look to see if I can replicate this - I am getting a null response on non-existant keys. Could you provide some reproduction steps?

[Huskydog9988]

If you want a fairly exact replication of our deployment, we are using this helm chart https://github.com/waldo-vision/waldo-helm/tree/main/charts/http-redis. The actual deployment is just this chart used as a dependency.

[hiett]

Thank you. I will work on replicating and push up a patch when I find the issue!

[hiett]

I've done some local work, and I have upstash/ratelimit working properly. The error seems to originate from the actual Redis connection being closed - there is an unhandled client error response case that I'm going to fix, but the actual Redis closing potentially could be more on the Redis going down / losing connection? I'm not quite sure - there obviously could be a lot of complexity in the setup of how it's talking to Redis.

Hopefully this new case I'm going to add will provide more insight to the underlying Redix error and why it might've closed

The code inside SRH automaticaly re-builds the pool when it goes down. So on the next request it will re-build the pool and attempt new connections. I will take a look at the Helm chart and see if I can reproduce within a K8s cluster and spot anything there.

[Huskydog9988]

Alright, thanks for really digging into this, I honestly can't express how grateful I am for this level of support. If you need anything else, just let me know.

[hiett]

No worries, I use this in production at my work so it's in my best interest to fix anything that comes up :)

I will leave this issue open until I improve the error handling for this error.

Thanks for using it!

[eric-burel]

Hi @hiett , similar issue here, while the Redis instance is working fine. I can provide non minimal repro on the Devographics monorepo, but I am not doing anything fancy:

• Run Redis, run SRH
• Query a key that do not exist yet (haven't tried with existing key)

Redis and SRH are running fine, but the underlying Redis instance is not hit. Perhaps I've messed up the Redis connection string?

    docker run \
    --rm \
    -p 8080:80 \
    --label srh \
    -e SRH_MODE=env \
    -e SRH_TOKEN=fake-dev-token \
    -e SRH_CONNECTION_STRING="redis://localhost:6379" \
    hiett/serverless-redis-http:latest

[hiett]

@eric-burel Thanks for reaching out - this seems to be a misconfiguration issue. In Docker, localhost refers to inside the container, not the host network - so your container is looking "internally" for 6379, rather than via the network.

There are a few simple solutions to this:

• Run redis & srh in a docker compose, whereby they're in the same Docker network, so you can address via the hostname, e.g. redis
• Use host.docker.internal if on an OS that supports it

You can read more about this networking situation with Docker at https://docs.docker.com/desktop/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host

For Kubernetes, if you are in the same namespace, you can just address via the service name. This is similar to the docker shared network approach.

The example for Docker Compose (https://github.com/hiett/serverless-redis-http#via-docker-compose) illustrates this approach

Hope this helps!

[eric-burel]

Long time since I used Docker sorry ':D works better now