ghost
commented
8 years ago Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):
This happened to us today. When it did, I saw things like this in the logs:
Logged from file models.py, line 64
ERROR:vr.common.models:Failed to connect to <Host ply16.pl.yougov.local>
Traceback (most recent call last):
File "/app/.heroku/venv/lib/python2.7/site-packages/vr/common/models.py", line 162, in _get_and_cache_procs
proc_list = _retry(3, self.supervisor.getAllProcessInfo)
File "/app/.heroku/venv/lib/python2.7/site-packages/vr/common/models.py", line 60, in _retry
return f(*args, **kwargs)
File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1301, in single_request
self.send_content(h, request_body)
File "/usr/lib/python2.7/xmlrpclib.py", line 1448, in send_content
connection.endheaders(request_body)
File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 797, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 778, in connect
self.timeout, self.source_address)
File "/app/.heroku/venv/lib/python2.7/site-packages/gevent/socket.py", line 86, in create_connection
raise err
timeout: timed out
Originally reported by: Anonymous
During some maintenance which required a host to be down for a little while (which remained a part of a swarm in the meantime), I noticed that the VR dashboard display would only show a few services if any.
Once the host came back, so did the full display of services.