search

robinhood / faust

Python Stream Processing
Other
6.72k stars 535 forks source link

Question about remote procedure calls #392

Open maertv opened 5 years ago

maertv commented 5 years ago

I have a question about remote procedure call (RPC) usage.

The test applications can be found here.

I have two (the number shouldn't matter) RPC apps:

These apps are meant to be called remotely by other apps.

For example, I have a test app that requires results from these apps.

This works but after about 5 minutes it starts to log following warning (every 7 seconds): [^-Consumer]: Possible livelock: COMMIT OFFSET NOT ADVANCING.

It still works but it creates a lot of log. In addition, it may mean that I have done something wrong which might cause even bigger problems down the line.

How should I make a call to remote agents?

I started the apps with following commands: 

python rpc-test-1.py worker --without-web
python rpc-test-2.py worker --without-web
python rpc-test-client.py worker --without-web

Versions

StephenSorriaux commented 5 years ago

Hi,

It seems like one (or more?) of your rpc_app.agent() are hitting the default timeout of 5min (see https://faust.readthedocs.io/en/latest/userguide/settings.html?highlight=livelock#broker-commit-livelock-soft-timeout) to commit offsets to Kafka.

I did not give it a try but, is your problem still present if you remove the topic declaration in your rpc_app.agent() (https://gist.github.com/maertv/7d6734c223192ec093c567d671c7f3ca#file-rpc-test-client-py-L16 and https://gist.github.com/maertv/7d6734c223192ec093c567d671c7f3ca#file-rpc-test-client-py-L24)?

maertv commented 5 years ago

Hi,

Thank you for your response.

Yes, it seems that I am hitting the default timeout.

I did not give it a try but, is your problem still present if you remove the topic declaration in your rpc_app.agent() (https://gist.github.com/maertv/7d6734c223192ec093c567d671c7f3ca#file-rpc-test-client-py-L16 and https://gist.github.com/maertv/7d6734c223192ec093c567d671c7f3ca#file-rpc-test-client-py-L24)?

I suspect that I am misunderstanding something because if I remove the topic declarations from these agents then the messages do not reach the RPC agents. It seems that without the topic declaration the agent will create a topic, for example RPCClientApp-rpc-test-client.rpc_test_1_noop, and the message will be published to that topic.

I tried using these autocreated topics also in RPC agents but the RPCClientApp continued to log [^-Consumer]: Possible livelock: COMMIT OFFSET NOT ADVANCING warnings.