Websocket connectivity between server and analytics

jonyrock-back commented 4 years ago

It's hard to support zmq on node.js side because of many reasons. last version zeromq.js supports only Node.js v.10.2+

all debug / production connectivity is removed and only websocket connection is available.

It fixes #753, #812 and #852 (closing port part)

Changes

Server

remove zeromq dependency from server/package.json
add dependency to websockets/ws and @types/ws into server/package.json
remove zmqConectionString: ZMQ_CONNECTION_STRING, from API info response (which has / url): not sure we should expose it anyway and it doesn't make sense since analytics connects to hastic server over http (same port as the API)
remove from config ZMQ_IPC_PATH, ZMQ_DEV_PORT, ZMQ_HOST vars
no ZMQ_IPC_PATH folder creation in data_service.ts
no npm rebuild zeromq in node 6.14 build process
server sends error if second instance of analytics tries to connect
add transform-object-rest-spread for transpiling the ws lib for node, read more about that

Analytics

add dependency to aaugustin/websockets in analytics/requrements.txt
remove ZMQ_DEV_PORT and ZMQ_CONNECTION_STRING and replace it with HASTIC_SERVER_URL

Notes

we still use pyzmq for messaging between threads/actors, see AsyncZmqThread
I encourage all supporters of this part to read comment in https://github.com/aaugustin/websockets/blob/master/src/websockets/client.py#L313 (and give a star to https://github.com/aaugustin/websockets)
Give a star to https://github.com/websockets/ws
Maybe we should send a webhock about bad analytic connect attempts
there are optional deps to https://github.com/websockets/bufferutil and utf-8-validate which I don't know what to do with.
the minumum supported version for the dev mode is 8.3.0 now. It's because of https://github.com/websockets/ws/issues/1703 (we don't use babel-loader in the dev mode)

Known issues

reconnecting in analytics to server on drop increases the number of ThreadPoolExecutor

rozetko commented 4 years ago

Tried to test it, but failed. There are at least 2 problems:

Connection to analytics drops when learning starts. Server logs after I click "Apply" in the panel:

Analytics is up
LEARNING started for abfcc8T9ZoRSrXpl
query time range: from Wed Feb 19 2020 13:41:32 GMT+0300 (MSK) to Wed Feb 19 2020 19:41:32 GMT+0300 (MSK)
run LEARN task, id:abfcc8T9ZoRSrXpl
Analytics is down
(node:4427) UnhandledPromiseRejectionWarning: Can`t send because analytics is not connected
(node:4427) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 6)

Learning starts though, this is what I see in analytics' logs when learning is finished:

2020-02-19 19:45:41,385 [Analytics] [DEBUG]  End correctly handle_analytic_task with anatytic unit: abfcc8T9ZoRSrXpl
2020-02-19 19:45:41,386 [Analytics] [ERROR]  Task exception was never retrieved
future: <Task finished coro=<ServerService._on_message_to_thread() done, 
defined at bin/../analytics/services/server_service.py:87> exception=AttributeError("'NoneType' object has no attribute 'send'",)>
Traceback (most recent call last):
File "bin/../analytics/services/server_service.py", line 88, in _on_message_to_thread
await self.__server_socket.send(message)
AttributeError: 'NoneType' object has no attribute 'send'

Tested with 2 node versions:

python3.6 --version
Python 3.6.7

node -v
v11.15.0

python3.6 --version
Python 3.6.7

node -v
v8.16.0

jonyrock-back commented 4 years ago

@rozetko was it dev or prod build?

rozetko commented 4 years ago

@jonyrock-back dev

jonyrock-back commented 4 years ago

Looks like I have this issue here: https://github.com/babel/babel-loader/issues/170

rozetko commented 4 years ago

More logs from analytics:

2020-03-04 20:57:30,477 [Analytics] [DEBUG]  client - event = eof_received()
2020-03-04 20:57:30,534 [Analytics] [DEBUG]  Start handle_analytic_task with analytic unit: nv3eCrznaqm2yhJ6
2020-03-04 20:57:30,536 [Analytics] [DEBUG]  client - event = connection_lost(None)
2020-03-04 20:57:30,536 [Analytics] [DEBUG]  Analytics get task with type: LEARN for unit: nv3eCrznaqm2yhJ6
2020-03-04 20:57:30,537 [Analytics] [DEBUG]  client - state = CLOSED
2020-03-04 20:57:30,553 [Analytics] [DEBUG]  client x code = 1005, reason = [no reason]
2020-03-04 20:57:30,558 [Analytics] [DEBUG]  Start method fit for analytic unit nv3eCrznaqm2yhJ6
2020-03-04 20:57:30,587 [Analytics] [ERROR]  Task exception was never retrieved
future: <Task finished coro=<ServerService._on_message_to_thread() done, 
defined at bin/../analytics/services/server_service.py:87> exception=ConnectionClosedError('code = 1005 (no status code [internal]), no reason',)>Traceback (most recent call last):
  File "bin/../analytics/services/server_service.py", line 88, in _on_message_to_thread
    await self.__server_socket.send(message)
  File "/home/rozetko/.local/lib/python3.6/site-packages/websockets/protocol.py", line 555, in send
    await self.ensure_open()
  File "/home/rozetko/.local/lib/python3.6/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1005 (no status code 
[internal]), no reason

hastic-zzz / hastic-server