OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Oracle Linux Server 7.7 Kernel: 4.14.35-1902.8.4.el7uek.x86_64
ElasticHQ Version: v3.5.12
Elasticsearch Version: 7.4.2 (using Open Distro for ElasticSearch 1.4.0)
Python version (ignore is using docker image): 3.6.8
Browser Vendor and Version (if applicable): Chrome 78.0.3904.108 (Official Build) (64 bit)
Issue Description
I installed and run the server on last Friday, and was able to see all the live metrics, today when i'm back to the office, application.py was stopped and i restarted it and i can't see now any data on Metrics.
I've tried to reinitialize the db, but it hasn't solved.
Logs
application.log
2020-04-06 14:01:57,379 INFO elastichq globals.get:115 Getting Cache Key: elastichq.service.HQService:get_settings|somecluster
2020-04-06 14:02:10,467 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet OPEN data {'sid': 'd2d7895485e447deae6df4533d9f4aa1', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
2020-04-06 14:02:10,467 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet MESSAGE data 0
2020-04-06 14:02:10,664 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet MESSAGE data 0/ws,
2020-04-06 14:02:10,664 INFO socketio server.emit:226 emitting event "event" to d2d7895485e447deae6df4533d9f4aa1 [/ws]
2020-04-06 14:02:10,665 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet MESSAGE data 2/ws,["event",{"connected":true}]
2020-04-06 14:02:10,665 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet MESSAGE data 0/ws
2020-04-06 14:02:10,680 INFO engineio socket.handle_get_request:101 d2d7895485e447deae6df4533d9f4aa1: Received request to upgrade to websocket
2020-04-06 14:02:10,708 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet MESSAGE data 2/ws,["join",{"room_name":"somecluster::nodes"}]
2020-04-06 14:02:10,708 INFO socketio server._handle_event:453 received event "join" from d2d7895485e447deae6df4533d9f4aa1 [/ws]
2020-04-06 14:02:10,708 DEBUG elastichq socket.joined:53 Received room join: {'room_name': 'somecluster::nodes'}
2020-04-06 14:02:10,708 INFO socketio server.enter_room:280 d2d7895485e447deae6df4533d9f4aa1 is entering room somecluster::nodes [/ws]
2020-04-06 14:02:10,709 ERROR engineio server._trigger_event:425 message handler error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/engineio/server.py", line 423, in _trigger_event
return self.handlersevent
File "/usr/local/lib/python3.6/site-packages/socketio/server.py", line 522, in _handle_eio_message
self._handle_event(sid, pkt.namespace, pkt.id, pkt.data)
File "/usr/local/lib/python3.6/site-packages/socketio/server.py", line 458, in _handle_event
self._handle_event_internal(self, sid, data, namespace, id)
File "/usr/local/lib/python3.6/site-packages/socketio/server.py", line 461, in _handle_event_internal
r = server._trigger_event(data[0], namespace, sid, data[1:])
File "/usr/local/lib/python3.6/site-packages/socketio/server.py", line 490, in _trigger_event
return self.handlers[namespace][event](args)
File "/usr/local/lib/python3.6/site-packages/flask_socketio/init.py", line 251, in _handler
args)
File "/usr/local/lib/python3.6/site-packages/flask_socketio/init.py", line 634, in _handle_event
ret = handler(args)
File "/usr/share/elasticsearch-HQ-master/elastichq/api/socket.py", line 60, in joined
print(rooms())
OSError: [Errno 5] Input/output error
2020-04-06 14:02:10,710 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet NOOP data None
2020-04-06 14:02:10,735 INFO engineio socket._websocket_handler:203 d2d7895485e447deae6df4533d9f4aa1: Upgrade to websocket successful
2020-04-06 14:02:20,416 INFO engineio socket.receive:52 216d67b88b2848998b703faa2c641209: Received packet PING data None
2020-04-06 14:02:20,416 INFO engineio socket.send:91 216d67b88b2848998b703faa2c641209: Sending packet PONG data None
2020-04-06 14:02:35,677 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet PING data None
2020-04-06 14:02:35,678 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet PONG data None
2020-04-06 14:02:45,433 INFO engineio socket.receive:52 216d67b88b2848998b703faa2c641209: Received packet PING data None
2020-04-06 14:02:45,433 INFO engineio socket.send:91 216d67b88b2848998b703faa2c641209: Sending packet PONG data None
2020-04-06 14:03:00,683 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet PING data None
2020-04-06 14:03:00,684 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet PONG data None
2020-04-06 14:03:10,449 INFO engineio socket.receive:52 216d67b88b2848998b703faa2c641209: Received packet PING data None
2020-04-06 14:03:10,450 INFO engineio socket.send:91 216d67b88b2848998b703faa2c641209: Sending packet PONG data None
2020-04-06 14:03:25,689 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet PING data None
2020-04-06 14:03:25,689 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet PONG data None
2020-04-06 14:03:35,466 INFO engineio socket.receive:52 216d67b88b2848998b703faa2c641209: Received packet PING data None
2020-04-06 14:03:35,467 INFO engineio socket.send:91 216d67b88b2848998b703faa2c641209: Sending packet PONG data None
browser errors that i've catched when metrics page is open.
angular.js:14791 Error: Err: unable to reach elastichq.org is not valid version string.
at m (index.js:81)
at e.value (footer.controller.js:40)
at footer.controller.js:22
at angular.js:17160
at angular.js:17208
at c.$digest (angular.js:18342)
at c.$apply (angular.js:18640)
at angular.js:12618
at w (angular.js:12862)
at XMLHttpRequest.b.onload (angular.js:12779) "Possibly unhandled rejection: {}"
(anonymous) @ angular.js:14791
General information
Issue Description
I installed and run the server on last Friday, and was able to see all the live metrics, today when i'm back to the office, application.py was stopped and i restarted it and i can't see now any data on Metrics. I've tried to reinitialize the db, but it hasn't solved.
Logs
application.log
2020-04-06 14:01:57,379 INFO elastichq globals.get:115 Getting Cache Key: elastichq.service.HQService:get_settings|somecluster 2020-04-06 14:02:10,467 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet OPEN data {'sid': 'd2d7895485e447deae6df4533d9f4aa1', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000} 2020-04-06 14:02:10,467 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet MESSAGE data 0 2020-04-06 14:02:10,664 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet MESSAGE data 0/ws, 2020-04-06 14:02:10,664 INFO socketio server.emit:226 emitting event "event" to d2d7895485e447deae6df4533d9f4aa1 [/ws] 2020-04-06 14:02:10,665 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet MESSAGE data 2/ws,["event",{"connected":true}] 2020-04-06 14:02:10,665 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet MESSAGE data 0/ws 2020-04-06 14:02:10,680 INFO engineio socket.handle_get_request:101 d2d7895485e447deae6df4533d9f4aa1: Received request to upgrade to websocket 2020-04-06 14:02:10,708 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet MESSAGE data 2/ws,["join",{"room_name":"somecluster::nodes"}] 2020-04-06 14:02:10,708 INFO socketio server._handle_event:453 received event "join" from d2d7895485e447deae6df4533d9f4aa1 [/ws] 2020-04-06 14:02:10,708 DEBUG elastichq socket.joined:53 Received room join: {'room_name': 'somecluster::nodes'} 2020-04-06 14:02:10,708 INFO socketio server.enter_room:280 d2d7895485e447deae6df4533d9f4aa1 is entering room somecluster::nodes [/ws] 2020-04-06 14:02:10,709 ERROR engineio server._trigger_event:425 message handler error Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/engineio/server.py", line 423, in _trigger_event return self.handlersevent File "/usr/local/lib/python3.6/site-packages/socketio/server.py", line 522, in _handle_eio_message self._handle_event(sid, pkt.namespace, pkt.id, pkt.data) File "/usr/local/lib/python3.6/site-packages/socketio/server.py", line 458, in _handle_event self._handle_event_internal(self, sid, data, namespace, id) File "/usr/local/lib/python3.6/site-packages/socketio/server.py", line 461, in _handle_event_internal r = server._trigger_event(data[0], namespace, sid, data[1:]) File "/usr/local/lib/python3.6/site-packages/socketio/server.py", line 490, in _trigger_event return self.handlers[namespace][event](args) File "/usr/local/lib/python3.6/site-packages/flask_socketio/init.py", line 251, in _handler args) File "/usr/local/lib/python3.6/site-packages/flask_socketio/init.py", line 634, in _handle_event ret = handler(args) File "/usr/share/elasticsearch-HQ-master/elastichq/api/socket.py", line 60, in joined print(rooms()) OSError: [Errno 5] Input/output error 2020-04-06 14:02:10,710 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet NOOP data None 2020-04-06 14:02:10,735 INFO engineio socket._websocket_handler:203 d2d7895485e447deae6df4533d9f4aa1: Upgrade to websocket successful 2020-04-06 14:02:20,416 INFO engineio socket.receive:52 216d67b88b2848998b703faa2c641209: Received packet PING data None 2020-04-06 14:02:20,416 INFO engineio socket.send:91 216d67b88b2848998b703faa2c641209: Sending packet PONG data None 2020-04-06 14:02:35,677 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet PING data None 2020-04-06 14:02:35,678 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet PONG data None 2020-04-06 14:02:45,433 INFO engineio socket.receive:52 216d67b88b2848998b703faa2c641209: Received packet PING data None 2020-04-06 14:02:45,433 INFO engineio socket.send:91 216d67b88b2848998b703faa2c641209: Sending packet PONG data None 2020-04-06 14:03:00,683 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet PING data None 2020-04-06 14:03:00,684 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet PONG data None 2020-04-06 14:03:10,449 INFO engineio socket.receive:52 216d67b88b2848998b703faa2c641209: Received packet PING data None 2020-04-06 14:03:10,450 INFO engineio socket.send:91 216d67b88b2848998b703faa2c641209: Sending packet PONG data None 2020-04-06 14:03:25,689 INFO engineio socket.receive:52 d2d7895485e447deae6df4533d9f4aa1: Received packet PING data None 2020-04-06 14:03:25,689 INFO engineio socket.send:91 d2d7895485e447deae6df4533d9f4aa1: Sending packet PONG data None 2020-04-06 14:03:35,466 INFO engineio socket.receive:52 216d67b88b2848998b703faa2c641209: Received packet PING data None 2020-04-06 14:03:35,467 INFO engineio socket.send:91 216d67b88b2848998b703faa2c641209: Sending packet PONG data None
browser errors that i've catched when metrics page is open.
websocket.js:110 WebSocket connection to 'ws://someserver:5000/socket.io/?EIO=3&transport=websocket&sid=206392aa4dbe47d99e3841ce7c9bddd1' failed: Received a broken close frame containing invalid UTF-8. h.doOpen @ websocket.js:110 o.open @ transport.js:83 u.probe @ socket.js:400 u.onOpen @ socket.js:421 u.onHandshake @ socket.js:482 u.onPacket @ socket.js:444 (anonymous) @ socket.js:276 r.emit @ index.js:133 o.onPacket @ transport.js:148 (anonymous) @ polling.js:144 (anonymous) @ browser.js:604 t.decodePayloadAsBinary @ browser.js:603 t.decodePayload @ browser.js:374 u.onData @ polling.js:148 (anonymous) @ polling-xhr.js:127 r.emit @ index.js:133 u.onData @ polling-xhr.js:300 u.onLoad @ polling-xhr.js:367 hasXDR.t.onreadystatechange @ polling-xhr.js:253 polling-xhr.js:265 GET http://someserver:5000/socket.io/?EIO=3&transport=polling&t=N5Ff0qo&sid=206392aa4dbe47d99e3841ce7c9bddd1 400 (BAD REQUEST)
angular.js:14791 Error: Err: unable to reach elastichq.org is not valid version string. at m (index.js:81) at e.value (footer.controller.js:40) at footer.controller.js:22 at angular.js:17160 at angular.js:17208 at c.$digest (angular.js:18342) at c.$apply (angular.js:18640) at angular.js:12618 at w (angular.js:12862) at XMLHttpRequest.b.onload (angular.js:12779) "Possibly unhandled rejection: {}" (anonymous) @ angular.js:14791