Client is not resilient to cognitive engine disconnects

[achanana]

The Gabriel ZeroMQ client raises the following exception when the cognitive engine disconnects from the Gabriel server. Instead, there should be a mechanism to allow the cognitive engine to be reconnected so that the client is stable.

INFO: Connecting to server at tcp://localhost:9099
INFO: Sent hello message to server
INFO: Received welcome from server
ERROR: Output status was: ENGINE_ERROR
ERROR: Task exception was never retrieved
future: <Task finished name='Task-1' coro=<ZeroMQClient._launch_helper() done, defined at /home/achanana/lab-work/gabriel/python-client/src/gabriel_client/zeromq_client.py:79> exception=Exception('No engine for source')>
Traceback (most recent call last):
  File "/home/achanana/lab-work/gabriel/python-client/src/gabriel_client/zeromq_client.py", line 98, in _launch_helper
    await asyncio.gather(*tasks)
  File "/home/achanana/lab-work/gabriel/python-client/src/gabriel_client/zeromq_client.py", line 144, in _consumer_handler
    self._process_response(to_client.response)
  File "/home/achanana/lab-work/gabriel/python-client/src/gabriel_client/zeromq_client.py", line 178, in _process_response
    raise Exception('No engine for source')
Exception: No engine for source

[teiszler]

Why is any engine disconnecting during normal operation? The only time I have seen them disconnect is if they crash, either in startup or when they are sent bad data that puts them into a bad state.

[achanana]

I intentionally disconnected an engine. In the future, when engines may not be on the same machine, or if they crash, they should not cause the client to crash as well.