faust v1.9.0 not working with sasl authentication

[jabonte]

Steps to reproduce

configure faust to use sasl

app = faust.App(

, broker=broker, key_serializer='json', value_serializer='json', broker_credentials=faust.SASLCredentials( username=, password=, ) ) ## Expected behavior That it works. (It works as expected with v1.8.0) ## Actual behavior program dies with a AssertionError during startup of the worker ## Full traceback [2019-11-06 14:27:33,119] [5519] [ERROR] [^Worker]: Error: AssertionError("yield from wasn't used with future",) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/mode/worker.py", line 267, in execute_from_commandline self.loop.run_until_complete(self._starting_fut) File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 727, in start await self._default_start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 734, in _default_start await self._actually_start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 758, in _actually_start await child.maybe_start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 786, in maybe_start await self.start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 727, in start await self._default_start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 734, in _default_start await self._actually_start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 758, in _actually_start await child.maybe_start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 786, in maybe_start await self.start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 727, in start await self._default_start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 734, in _default_start await self._actually_start() File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 751, in _actually_start await self.on_start() File "/usr/local/lib/python3.6/site-packages/faust/transport/drivers/aiokafka.py", line 742, in on_start await producer.start() File "/usr/local/lib/python3.6/site-packages/aiokafka/producer/producer.py", line 171, in start await self.client.bootstrap() File "/usr/local/lib/python3.6/site-packages/aiokafka/client.py", line 203, in bootstrap version_hint=version_hint) File "/usr/local/lib/python3.6/site-packages/aiokafka/conn.py", line 90, in create_conn await conn.connect() File "/usr/local/lib/python3.6/site-packages/aiokafka/conn.py", line 214, in connect await self._do_sasl_handshake() File "/usr/local/lib/python3.6/site-packages/aiokafka/conn.py", line 281, in _do_sasl_handshake payload, expect_response = res AssertionError: yield from wasn't used with future # Versions CPython 3.6.8 (Linux x86_64) faust=1.9.0 aiokafka=1.1.3 aiohttp=3.6.2 linux=3.10.0-957.27.2.el7.x86_64

[gvdmarck]

Same here. A totally internal error (we only upgraded our packages) not even being responded/acknowledged in 3 months ? What is this ?

Full traceback:

Traceback (most recent call last): File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/worker.py", line 273, in execute_from_commandline self.loop.run_until_complete(self._starting_fut) File "/opt/conda/envs/app/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete return future.result() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 736, in start await self._default_start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 743, in _default_start await self._actually_start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 767, in _actually_start await child.maybe_start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 795, in maybe_start await self.start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 736, in start await self._default_start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 743, in _default_start await self._actually_start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 767, in _actually_start await child.maybe_start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 795, in maybe_start await self.start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 736, in start await self._default_start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 743, in _default_start await self._actually_start() File "/opt/conda/envs/app/lib/python3.7/site-packages/mode/services.py", line 760, in _actually_start await self.on_start() File "/opt/conda/envs/app/lib/python3.7/site-packages/faust/transport/drivers/aiokafka.py", line 760, in on_start await producer.start() File "/opt/conda/envs/app/lib/python3.7/site-packages/aiokafka/producer/producer.py", line 171, in start await self.client.bootstrap() File "/opt/conda/envs/app/lib/python3.7/site-packages/aiokafka/client.py", line 203, in bootstrap version_hint=version_hint) File "/opt/conda/envs/app/lib/python3.7/site-packages/aiokafka/conn.py", line 90, in create_conn await conn.connect() File "/opt/conda/envs/app/lib/python3.7/site-packages/aiokafka/conn.py", line 214, in connect await self._do_sasl_handshake() File "/opt/conda/envs/app/lib/python3.7/site-packages/aiokafka/conn.py", line 281, in _do_sasl_handshake payload, expect_response = res RuntimeError: await wasn't used with future

Versions

CPython 3.7.6 (Linux x86_64) faust=1.10.1 aiokafka=1.1.3 aiohttp=3.6.2 linux=3.10.0-957.27.2.el7.x86_64

[gvdmarck]

Is robinhood still maintainng faust ? Or is this repo dead ? The fix to this issue is to update your aiokafka lib so that it is in sync with the apache one, the pull request has been there for 3 months : https://github.com/robinhood/aiokafka/pull/18

[hakwerk]

I can confirm that this issue still exists in Faust 1.10.2 and that the pull request fixes it. I applied the fixes from the PR into my virtualenv and it solves the issue.

[gvdmarck]

@ask ? Can we also have a comment on the faust follow-up status ?

[gvdmarck]

Even after emailing Robinhood directly, still no reactions what so ever.

Time to change client libraries for our applications. Does any one have returns on usage of python-kafka, pykafka or confluent-kafka ?

[deed02392]

I don't think it's a big enough of a deal to switch libraries for me. Is there a PR to fix this? Maybe just maintain your own forked package with the fix in it?

[ask]

aiokafka 1.1.6 is on PyPI with the fix

If anyone here is willing to help triage issues that would be a huge help to surface fixes for important issues sooner.

[chase2981]

I would definitely strongly consider doing my part in helping maintain this repo seems it is impressive to all who come across it; however, from what I gather this repo/package is no longer maintained. (https://github.com/robinhood/faust/pull/701#issuecomment-769354390 | https://github.com/robinhood/faust/issues/468#issuecomment-579686892) If this is true, which shouldn't even be an uncertainty, but assuming this is true, then the least we could do is put up a "NO LONGER MAINTAINED" banner on the top of the docs and the README.md, so that way others don't go down this same path thinking this is actually a legit solution only to later discover nobody is actively facilitating the proper maintenance of it.

[chase2981]

For my company, the very fact that this repo's maintenance is questionable is a huge red flag. We are currently trying to get away from another open source kinesis streaming lib where the same thing happened, it became un-maintained a few years back and now we simply will not tolerate moving to any streaming lib whose maintenance is even somewhat questionable. (https://github.com/LeoPlatform) Our entire organization's data is on the line. It's been a huge hassle across our entire organization. That's why we were moving to kafka and why we started looking into faust in the first place.