search

robinhood / faust

Python Stream Processing
Other
6.72k stars 535 forks source link

Unable to force application to exit if it fails to make a connection to a bootstrap server #711

Open reynoldsm88 opened 3 years ago

reynoldsm88 commented 3 years ago

Checklist

Faust version: 1.10.4

Steps to reproduce

Expected behavior

You should be able to do one of the following:

  1. Tell the application to automatically exit if it fails to make a connection to a Kafka broker
  2. Expose some kind of lifecycle event listener that an application can easily plug into in order to implement desired shutdown/recovery behavior

Actual behavior

Faust fails to make a connection and then hangs. This seems like it might be similar/related to https://github.com/robinhood/faust/issues/484

Full traceback

[2021-05-04 14:52:46,278] [1] [ERROR] Unable connect to "172.31.68.228:19092": [Errno 111] Connect call failed ('172.31.68.228', 19092)
[2021-05-04 14:52:46,287] [1] [ERROR] [^Worker]: Error: ConnectionError("Unable to bootstrap from [('172.31.68.228', 19092, <AddressFamily.AF_INET: 2>)]",)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/mode/worker.py", line 273, in execute_from_commandline
    self.loop.run_until_complete(self._starting_fut)
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 736, in start
    await self._default_start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 743, in _default_start
    await self._actually_start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 767, in _actually_start
    await child.maybe_start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 795, in maybe_start
    await self.start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 736, in start
    await self._default_start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 743, in _default_start
    await self._actually_start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 767, in _actually_start
    await child.maybe_start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 795, in maybe_start
    await self.start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 736, in start
    await self._default_start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 743, in _default_start
    await self._actually_start()
  File "/usr/local/lib/python3.6/site-packages/mode/services.py", line 760, in _actually_start
    await self.on_start()
  File "/usr/local/lib/python3.6/site-packages/faust/transport/drivers/aiokafka.py", line 1021, in on_start
    await producer.start()
  File "/usr/local/lib/python3.6/site-packages/aiokafka/producer/producer.py", line 178, in start
    await self.client.bootstrap()
  File "/usr/local/lib/python3.6/site-packages/aiokafka/client.py", line 231, in bootstrap
    'Unable to bootstrap from {}'.format(self.hosts))
kafka.errors.ConnectionError: ConnectionError: Unable to bootstrap from [('172.31.68.228', 19092, <AddressFamily.AF_INET: 2>)]
+ƒaµS† v1.10.4+------------------------------------------+
| id          | sofia                                    |
| transport   | [URL('kafka://123.456.7.89:19092')]     |
| store       | memory:                                  |
| web         | http://localhost:6066/                   |
| log         | -stderr- (warn)                          |
| pid         | 1                                        |
| hostname    | ip-172-31-68-228.ec2.internal            |
| platform    | CPython 3.6.8 (Linux x86_64)             |
| drivers     |                                          |
|   transport | aiokafka=1.1.6                           |
|   web       | aiohttp=3.7.4.post0                      |
| datadir     | /sofia-data                              |
| appdir      | /sofia-data/v1                           |
+-------------+------------------------------------------+

### application hangs...

Versions

reynoldsm88 commented 3 years ago

This issue is still present, would like to hear about a resolution

reynoldsm88 commented 3 years ago

I suppose this might be related to: https://github.com/robinhood/faust/issues/484. However, the answer related to Docker does not seem acceptable to me. For a framework that is designed to implement potentially long running applications, I would expect that the framework itself provides some sort of introspection into the lifecycle so that the application developer can implement domain specific:

Using faust within Docker? If so, try with https://github.com/Yelp/dumb-init

basically change your entrypoint to use dumb-init: ENTRYPOINT ["/usr/bin/dumb-init", "/opt/start-my-awesome-faust-job.sh"]