search

samuelcolvin / arq

Fast job queuing and RPC in python with asyncio and redis.
https://arq-docs.helpmanual.io/
MIT License
1.98k stars 165 forks source link

Allow for user specified Redis key prefix #436

Open ipmb opened 2 months ago

ipmb commented 2 months ago

In multi-tenant scenarios, Redis recommends using namespaces to segment tenants. This is especially useful for non-production environments with hosted Redis providers. Instead of paying for a new Redis instance for each environment, the key space can be segmented and assigned to an environment.

This issue is to request the ability to define an arbitrary key prefix or namespace that would allow 2 arq services to operate on a single Redis instance/cluster.

Some prior art:

rossmacarthur commented 2 months ago

Could you explain why the queue_name option is not sufficient?

ipmb commented 2 months ago

The prefix would need to be applied to every key in Redis. It's not clear to me that queue_name will be used for all these values?

https://github.com/samuelcolvin/arq/blob/ab2dda2011ab27007650c4918d3704f3bf7ac13d/arq/constants.py#L1-L6

epicwhale commented 2 months ago

+1 I faced the same issue too when having two separate worker codebases use the same redis db. The queue name prefix applied to only a small subset of keys generated. So it was hard for me even while using a redis gui browser (like redis insights) to figure out which entries belonged to which. A clean name spacing separation at the root level would be more ideal.