apache / openwhisk-package-kafka

Apache OpenWhisk package for communicating with Kafka or Message Hub
https://openwhisk.apache.org/
Apache License 2.0
33 stars 43 forks source link

Catch Doctor exceptions and do not persist consumer database connections #343

Closed dubee closed 5 years ago

dubee commented 5 years ago

An exception thrown when the Doctor is restarting a consumer will result in the Doctor completely shutting down due to the exception not being caught. As a result, exceptions in the Doctor's run loop need to be handled properly.

The exception causing the Doctor to crash occurs when the Doctor restarts a consumer and an attempt to create a database connection fails on consumer init. Since a presistant database connection is not needed by the consumer itself, only create the database connection when it is needed to automatically disable a trigger.

Below is a stacktrace showing the described exception occuring:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/KafkaFeedProvider/thedoctor.py", line 72, in run
    consumer.restart()
  File "/KafkaFeedProvider/consumer.py", line 119, in restart
    self.process = ConsumerProcess(self.trigger, self.params, self.sharedDictionary)
  File "/KafkaFeedProvider/consumer.py", line 182, in __init__
    self.database = Database()
  File "/KafkaFeedProvider/database.py", line 48, in __init__
    self.client.connect()
  File "/usr/local/lib/python2.7/site-packages/cloudant/client.py", line 117, in connect
    self.session_login(self._user, self._auth_token)
  File "/usr/local/lib/python2.7/site-packages/cloudant/client.py", line 173, in session_login
    headers={'Content-Type': 'application/x-www-form-urlencoded'}
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 518, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/cloudant/_common_util.py", line 310, in request
    method, url, timeout=self._timeout, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='OMITTED', port=443): Max retries exceeded with url: /_session (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f84d75789d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))