[Bug - Investigation]: Redis url is 25 rps vs. using redis_port, redis_host, etc. (100 rps)

krrishdholakia commented 6 months ago

What happened?

When using the redis async client for usage-based-routing-v2, when initializing the redis cache via 'redis_url' it's significantly slower than using redis_port, host, etc.

router_settings:
  routing_strategy: usage-based-routing-v2 
  # redis_url: "os.environ/REDIS_URL"
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD

	usage-based-routing-v2 w/ redis ssl	usage-based-routing-v2 w/ redis host name, port, etc.	simple-shuffle w/ no redis
RPS	25.4	109.7	161.6

Relevant log output

No response

Twitter / LinkedIn details

No response

krrishdholakia commented 6 months ago

there's no perf impact caused b/c of ssl: Just tested with redis ssl as a env param vs. in the url

ishaan-jaff commented 6 months ago

We see this on our load testing too - the last stable version for using redis url was https://github.com/BerriAI/litellm/releases/tag/v1.35.15

jkolx commented 5 months ago

Possible reason: connection_pool not set properly

REDIS: ignoring argument: connection_pool. Not an allowed async_redis.Redis.from_url arg.

krrishdholakia commented 5 months ago

@jkolx what would the correct way to pass connection pool be? redis doesn't seem to support it with from_url

jkolx commented 5 months ago

Exactly that, you can't. The classmethod creates it's own pool in Redis client. However, you can forward pool creation arguments via the **kwargs.

krrishdholakia commented 5 months ago

worth testing - do you have code for how we should set this instead @jkolx

jkolx commented 5 months ago

Right off the bat I see the pool behavior is different, depending solely on how we instantiate the client. In redis-server --loglevel debug logs:

with REDIS_URL

redis-1    | 1:M 23 May 2024 16:25:12.766 - Accepted 192.168.0.3:53450
redis-1    | 1:M 23 May 2024 16:25:12.766 - Client closed connection id=4 addr=192.168.0.3:53450 laddr=192.168.0.2:6379 fd=9 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=16386 argv-mem=0 multi-mem=0 rbs=16392 rbp=16392 obl=0 oll=0 omem=0 tot-mem=33560 events=r cmd=ping user=default redir=-1 resp=2 lib-name=redis-py lib-ver=5.0.0
redis-1    | 1:M 23 May 2024 16:25:16.279 . 1 clients connected (0 replicas), 1101384 bytes in use

with REDIS_HOST/REDIS_PORT

redis-1    | 1:M 23 May 2024 16:26:03.914 - Accepted 192.168.0.3:50910
redis-1    | 1:M 23 May 2024 16:26:07.492 . 2 clients connected (0 replicas), 1103776 bytes in use

I haven't found the exact args needed to replicate the pool setup yet. Maybe this is something you can investigate further.

krrishdholakia commented 5 months ago

Curious - how're you using litellm today? @jkolx

matthew-walters commented 4 months ago

Using redis.Redis.from_url() creates a client with a ConnectionPool, whereas using redis.Redis() creates a client with a BlockingConnectionPool

BerriAI / litellm