`ResultTimeout` with Redis

[isaacleeai]

Issues

GitHub issues are for bugs. If you have questions, please ask them on the discussion board.

Checklist

• [x] Does your title concisely summarize the problem?
• [x] Did you include a minimal, reproducible example?
• [x] What OS are you using?
• [x] What version of Dramatiq are you using?
• [x] What did you do?
• [x] What did you expect would happen?
• [x] What happened?

What OS are you using?

CentOs7

What version of Dramatiq are you using?

1.5.0

What did you do?

called function completes and returns a list successfully ( I have chekd this with print statement, and it prints correctly whenever I call the function asynchronously ), but when I try to retrieve the retruned result with tacotron.send(model).get_result(backend = result_backend, block=True), I get:

Traceback (most recent call last):
  File "actor.py", line 55, in <module>
    ret = tacotron.send().get_result(backend = result_backend, block=True)
  File "/home/msl/.virtualenvs/venv_tts/lib/python3.5/site-packages/dramatiq-1.5.0-py3.5.egg/dramatiq/message.py", line 147, in get_result
    return backend.get_result(self, block=block, timeout=timeout)
  File "/home/msl/.virtualenvs/venv_tts/lib/python3.5/site-packages/dramatiq-1.5.0-py3.5.egg/dramatiq/results/backends/redis.py", line 81, in get_result
    raise ResultTimeout(message)
dramatiq.results.errors.ResultTimeout: tacotron()

this is how I defined tacotron(model):

@dramatiq.actor
def tacotron(model):
    return model.GetMel("President Trump met with other leaders at the Group of 20 conference.", 0)

I have put print statement inside GetMel of model, and it prints correctly -- i.e. right before it returns, I check the content of the list that is being returned, and they are fine. It prints within 1 second of the request.

Result backend: Redis I'm not sure if this is correct. How I installed and started Redis:

sudo pip install redis
sudo yum install redis
sudo systemctl start redis

Broker: RabbitMQ

What did you expect would happen?

for tacotron.send(model).get_result(backend = result_backend, block=True) to return successfully

What happened?

get_result(block=True) hung then errored

Traceback (most recent call last):
  File "actor.py", line 55, in <module>
    ret = tacotron.send().get_result(backend = result_backend, block=True)
  File "/home/msl/.virtualenvs/venv_tts/lib/python3.5/site-packages/dramatiq-1.5.0-py3.5.egg/dramatiq/message.py", line 147, in get_result
    return backend.get_result(self, block=block, timeout=timeout)
  File "/home/msl/.virtualenvs/venv_tts/lib/python3.5/site-packages/dramatiq-1.5.0-py3.5.egg/dramatiq/results/backends/redis.py", line 81, in get_result
    raise ResultTimeout(message)
dramatiq.results.errors.ResultTimeout: tacotron()

[Bogdanp]

Looks like you might not have set up your result backend correctly. Did you add the results middleware to your broker?

[isaacleeai]

I do so as instructed in your instructions:

from dramatiq.brokers.rabbitmq import RabbitmqBroker
from dramatiq.results.backends.redis import RedisBackend
from dramatiq.results import Results

result_backend = RedisBackend()

broker = RabbitmqBroker()
broker.add_middleware(Results(backend=result_backend))
dramatiq.set_broker(broker)

then, in the main of the same file:

if __name__ == '__main__':
    start = time.time()
    print("start time: " + str(start))
    ret = tacotron.send().get_result(block=True)

    print(type(ret))
    d = time.time() - start
    print(d)

However, it hangs and raises error in this line ret = tacotron.send().get_result(block=True)

[Bogdanp]

And you're running your workers in the background before running main, right?

Try increasing the timeout .get_result(block=True, timeout=900000).

[Bogdanp]

If that doesn't help, then upload the repo somewhere and I'll take a look later today.

[isaacleeai]

I'm not sure if that will fix. I checked by putting a print statement inside the function that I call, and it takes abut 0.5 seconds for the function to complete. However, even if it does complete, the returned value from the function never gets to my client.

Are you planning on running the code? Because the function I call requires some large dependency and some sensitive code from our company. Is it okay if I just send you parts that are relevent?

[Bogdanp]

Yeah, you can just e-mail me the relevant bits if you want.

[isaacleeai]

Thanks a lot. I'll triple-check before I send you code and if I still can't figure out, I'll send you via your email.

[isaacleeai]

Update: I have undone all the changes to the source code, and replaced a call to my library to a much simpler one:

@dramatiq.actor
def test_basic(model):
    print("hello from test_basic")
    t = "return this"
    return t

Still, the error persists. From dramatiq server console, I can see that "hello from test_basic" is printed, but I still get dramatiq.results.errors.ResultTimeout: test_basic()

Could it be because I am using virtual envirionment and my redis-server is located in /usr/bin/redis-server and rabbitmq-server is in /usr/sbin/rabbitmq-server while dramatiq is in ~/.virtualenvs/venv_tts/bin/dramatiq and the dramatiq library is in /home/msl/.virtualenvs/venv_tts/lib?

I made sure to donwload all dependencies with sudo patㅗ/to/venv/python -m pip install -e '.[all]' inside dramatiq source directory.

[isaacleeai]

update: I was able to reproduce the bug with minimal code.

setup

started from scratch on a different computer ( CentOS7 with Python3.5 ), I have installed dramatiq with exactly the following commands:

sudo /usr/bin/python3.5 -m pip install -e '.[all]'
sudo yum install redis
sudo yum install rabbitmq-server

test code

Then I wrote two files: actor.py -- used to start worker -- and request.py

import time
import dramatiq
from dramatiq.brokers.rabbitmq import RabbitmqBroker
from dramatiq.results.backends.redis import RedisBackend
from dramatiq.results import Results

result_backend = RedisBackend()

broker = RabbitmqBroker()
broker.add_middleware(Results(backend=result_backend))
dramatiq.set_broker(broker)

@dramatiq.actor
def test_basic(model):
    print("hello from test_basic")
    t = "return this"
    return t

And request.py which sends a simple request.

# request.py
import actor
import time

if __name__ == '__main__':
    start = time.time()
    print("start time: " + str(start))
    ret = actor.test_basic.send().get_result(block=True)
    print(ret)
    d = time.time() - start
    print(d)

test

Then I started rabitmq and reids, then booted up worker with dramatiq binary and sent a request as below:

sudo systemctl start rabbitmq-server
sudo systemctl start redis
dramatiq -p 1 -t 1 actor
python3.5 request.py

result

On the terminal window with the worker, I clearly see the printed message: "hello from test_basic"

but I get this error on the terminal that called request.py:

Traceback (most recent call last):
  File "request.py", line 7, in <module>
    ret = actor.test_basic.send().get_result(block=True)
  File "/usr/local/lib/python3.5/site-packages/dramatiq/message.py", line 147, in get_result
    return backend.get_result(self, block=block, timeout=timeout)
  File "/usr/local/lib/python3.5/site-packages/dramatiq/results/backends/redis.py", line 81, in get_result
    raise ResultTimeout(message)
dramatiq.results.errors.ResultTimeout: test_basic()

Thanks.

[Bogdanp]

Ah, sorry for not noticing this earlier. Your issue is that you haven't told the actor to store its results.

@dramatiq.actor
def test_basic(model):
    ...

should be

@dramatiq.actor(store_results=True)
def test_basic(model):
    ...

This is shown in the docs, but it's not really emphasized. Sorry!

[isaacleeai]

Oh, that's embarrassing haha. Thanks a lot! Saved me my day.

[satheeshkatipomu]

This discussion saved my time.

[Bogdanp]

I'm happy to take pointers re. what the documentation could be doing better to help in these cases.

[satheeshkatipomu]

may be need to highlight in get_results and wait methods in documentation

[denny4nl]

This discussion is really helpful to save me time.