Cluster client should retry gracefully for a reasonable amount of time when frontend server is stopped/restarted. I restarted our frontend server, with perhaps a 1 minute time gap between stop and start, and every client failed and needed to be manually restarted.
(env)ryan@bdplab0:~/git/cstar_perf$ supervisorctl status
crashmail RUNNING pid 2085, uptime 39 days, 20:42:36
cstar_perf_client FATAL Exited too quickly (process log may have details)
(env)ryan@bdplab0:~/git/cstar_perf$ supervisorctl tail cstar_perf_client
cent call last):
File "/home/ryan/git/cstar_perf/env/bin/cstar_perf_client", line 9, in <module>
load_entry_point('cstar-perf.frontend==1.0', 'console_scripts', 'cstar_perf_client')()
File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 821, in main
job_runner.run()
File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 203, in run
self.__ws_client.connect()
File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 70, in connect
ws.connect(self.ws_endpoint)
File "/home/ryan/git/cstar_perf/env/local/lib/python2.7/site-packages/websocket/_core.py", line 441, in connect
raise err
socket.error: [Errno 111] Connection refused
DEBUG:cstar_perf.client:Connecting to ws://cstar.datastax.com/api/cluster_comms ...
Traceback (most recent call last):
File "/home/ryan/git/cstar_perf/env/bin/cstar_perf_client", line 9, in <module>
load_entry_point('cstar-perf.frontend==1.0', 'console_scripts', 'cstar_perf_client')()
File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 821, in main
job_runner.run()
File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 203, in run
self.__ws_client.connect()
File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 70, in connect
ws.connect(self.ws_endpoint)
File "/home/ryan/git/cstar_perf/env/local/lib/python2.7/site-packages/websocket/_core.py", line 441, in connect
raise err
socket.error: [Errno 111] Connection refused
Cluster client should retry gracefully for a reasonable amount of time when frontend server is stopped/restarted. I restarted our frontend server, with perhaps a 1 minute time gap between stop and start, and every client failed and needed to be manually restarted.