DragonflyDb not working with python dramatiq workers

[vendari12]

Describe the bug I am running dragonflydb as our preferred cache in place of redis, everything works as expected until we introduced dramatiq task workers as we'd wanted to relief the longer tasks to a seperate process. The task workers started failing with this error redis.exceptions.ResponseError: Error running script (call to 976fb89fbb3c404c8c7bed59060c56521eb9ba69): @user_script:59: script tried accessing undeclared key, key: dramatiq:__heartbeats__

To Reproduce Steps to reproduce the behavior:

1. Run the docker.dragonflydb.io/dragonflydb/dragonfly docker image
2. Install and spin up dramatiq task workers dramatiq -p 4 -t 4 actors connected to the dragonfly instance

Expected behavior I expect the worker to start up with it specified configurations, N/B this works on Redis

Stack trace: ado-general-worker | [2024-10-22 13:05:37,328] [PID 27] [Thread-4] [dramatiq.worker.ConsumerThread(general.DQ)] [INFO] Restarting consumer in 3.00 seconds. trado-general-worker | [2024-10-22 13:05:37,328] [PID 26] [Thread-3] [dramatiq.worker.ConsumerThread(general)] [CRITICAL] Consumer encountered an unexpected error. trado-general-worker | Traceback (most recent call last): trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/brokers/redis.py", line 344, in __next__ trado-general-worker | data = self.message_cache.pop(0) trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | IndexError: pop from empty list trado-general-worker | trado-general-worker | During handling of the above exception, another exception occurred: trado-general-worker | trado-general-worker | Traceback (most recent call last): trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/worker.py", line 265, in run trado-general-worker | for message in self.consumer: trado-general-worker | ^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/brokers/redis.py", line 363, in __next__ trado-general-worker | self.message_cache = messages = self.broker.do_fetch( trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/brokers/redis.py", line 281, in do_dispatch trado-general-worker | return dispatch(args=args, keys=keys) trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/commands/core.py", line 5931, in __call__ trado-general-worker | return client.evalsha(self.sha, len(keys), *args) trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/commands/core.py", line 5307, in evalsha trado-general-worker | return self._evalsha("EVALSHA", sha, numkeys, *keys_and_args) trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/commands/core.py", line 5291, in _evalsha trado-general-worker | return self.execute_command(command, sha, numkeys, *keys_and_args) trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 508, in execute_command trado-general-worker | return conn.retry.call_with_retry( trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/retry.py", line 46, in call_with_retry trado-general-worker | return do() trado-general-worker | ^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 509, in <lambda> trado-general-worker | lambda: self._send_command_parse_response( trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 485, in _send_command_parse_response trado-general-worker | return self.parse_response(conn, command_name, **options) trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 525, in parse_response trado-general-worker | response = connection.read_response() trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^ trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/connection.py", line 516, in read_response trado-general-worker | raise response trado-general-worker | redis.exceptions.ResponseError: Error running script (call to 976fb89fbb3c404c8c7bed59060c56521eb9ba69): @user_script:59: script tried accessing undeclared key, key: dramatiq:__heartbeats__

Environment (please complete the following information):

• OS: [ubuntu 20.04]
• Containerized?: [Docker, Docker Compose]
• Dragonfly Version: [latest]

Reproducible Code Snippet

@dramatiq.actor(queue_name="usermgmt")
async def foo(
    args ...
):

[romange]

it's not really a reproducible example. We are not familiar with dramatiq framework. We need help to reproduce it, and step by step instructions. Best thing - to have a docker that we can just run against Dragonfly.

if dramatiq works with Redis cluster - you most likely can solve this by following the same guidelines like here: https://www.dragonflydb.io/docs/integrations/bullmq

[dranikpg]

It looks like they know their key names, they just don't bother to declare them https://github.com/Bogdanp/dramatiq/blob/master/dramatiq/brokers/redis/dispatch.lua

[romange]

Yep, but even without them, if namespace is wrapped with tags like this {mynamespace} and dragonfly runs with --lock_on_hashtags this should also work. https://github.com/Bogdanp/dramatiq/blob/master/dramatiq/brokers/redis/dispatch.lua#L21

[vendari12]

@romange I read about the performance concerns of using this --default_lua_flags=allow-undeclared-keys, does it only affect specific workloads or the entire system and also i can't seem to find the ENV flag for it with docker