search

JarbasHiveMind / HiveMind-core

Join the OVOS collective, utils for OpenVoiceOS mesh networking
Apache License 2.0
95 stars 14 forks source link

When listener restart, satellite is not able to reconnect #78

Open goldyfruit opened 7 months ago

goldyfruit commented 7 months ago

When listener restart, satellite is not able to reconnect, the only way is to wait either 20 minutes or restart the satellite.

ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener 2023-12-06 22:44:31.047 - HiveMind - hivemind_core.service:open:114 - INFO - authorizing client: VoiceSatelliteV0.3.0
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener Uncaught exception GET /?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6NjcyYWJhZDIyNWI2MjRjYTBlZjA1MjY5NmE1MTliMDE= (192.168.100.137)
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener HTTPServerRequest(protocol='http', host='192.168.100.54:5678', method='GET', uri='/?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6NjcyYWJhZDIyNWI2MjRjYTBlZjA1MjY5NmE1MTliMDE=', version='HTTP/1.1', remote_ip='192.168.100.137')
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener Traceback (most recent call last):
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 154, in decrypt_bin
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener     return cipher.decrypt_and_verify(ciphertext, tag)
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 567, in decrypt_and_verify
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener     self.verify(received_mac_tag)
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 508, in verify
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener     raise ValueError("MAC check failed")
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener ValueError: MAC check failed
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener 
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener During handling of the above exception, another exception occurred:
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener 
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener Traceback (most recent call last):
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/tornado/websocket.py", line 630, in _run_callback
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener     result = callback(*args, **kwargs)
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener              ^^^^^^^^^^^^^^^^^^^^^^^^^
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/service.py", line 107, in on_message
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener     message = self.client.decode(message)
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/protocol.py", line 103, in decode
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener     payload = decrypt_bin(self.crypto_key, payload)
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 156, in decrypt_bin
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener     raise DecryptionKeyError
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener hivemind_bus_client.exceptions.DecryptionKeyError
ovos-core-86bc9bc566-xzpvp ovos-hivemind-listener 2023-12-06 22:44:31.196 - HiveMind - hivemind_core.service:on_close:153 - INFO - disconnecting client: VoiceSatelliteV0.3.0:192.168.100.137::84a2872b-796c-43b1-b0f5-d5025cd0ca99
goldyfruit commented 6 months ago

Adding this maybe it can help https://forum.pydio.com/t/doc-password-encrypted-encryption-key-how-to-decrypt-it-using-the-password/3123/5

goldyfruit commented 6 months ago

Satellite log:

2024-02-01 11:48:04.923 - HiveMind-voice-sat - ovos_bus_client.client.client:on_error:120 - ERROR - === WebSocketTimeoutException('Connection timed out') ===
Traceback (most recent call last):
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_socket.py", line 115, in recv
    bytes_ = _recv()
             ^^^^^^^
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_socket.py", line 94, in _recv
    return sock.recv(bufsize)
           ^^^^^^^^^^^^^^^^^^
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_app.py", line 500, in setSock
    dispatcher.read(self.sock.sock, read, check)
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_app.py", line 88, in read
    if not read_callback():
           ^^^^^^^^^^^^^^^
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_app.py", line 515, in read
    op_code, frame = self.sock.recv_data_frame(True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_core.py", line 437, in recv_data_frame
    frame = self.recv_frame()
            ^^^^^^^^^^^^^^^^^
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_core.py", line 478, in recv_frame
    return self.frame_buffer.recv_frame()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_abnf.py", line 363, in recv_frame
    self.recv_header()
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_abnf.py", line 319, in recv_header
    header = self.recv_strict(2)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_abnf.py", line 398, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_core.py", line 563, in _recv
    return recv(self.sock, bufsize)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goldyfruit/.venvs/ovos/lib/python3.11/site-packages/websocket/_socket.py", line 117, in recv
    raise WebSocketTimeoutException("Connection timed out")
websocket._exceptions.WebSocketTimeoutException: Connection timed out
2024-02-01 11:48:04.948 - HiveMind-voice-sat - ovos_bus_client.client.client:on_error:127 - ERROR - Exception closing websocket at ws://10.17.3.102:5678?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk=: Connection timed out
2024-02-01 11:48:04.954 - HiveMind-voice-sat - ovos_bus_client.client.client:on_error:129 - WARNING - Message Bus Client will reconnect in 5.0 seconds.
2024-02-01 11:48:10.006 - HiveMind-voice-sat - ovos_bus_client.client.client:on_open:93 - DEBUG - Connected
2024-02-01 11:48:10.018 - HiveMind-voice-sat - hivemind_bus_client.client:emit:279 - DEBUG - sending to HiveMind: HiveMessageType.BUS
2024-02-01 11:48:10.123 - HiveMind-voice-sat - hivemind_bus_client.protocol:handle_hello:129 - INFO - HELLO: {'pubkey': '-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nxsFNBGW3yEIBEAC5n/NGeSNmTOlSBuZS5U3+0HH8wGTeqsXSuMGv+1XWjbkI5ALU\nSyNMrZBUhtvDtDc1QYx5i0qA26anUbCCOVuRvxFOT1IzjLhAg1hRGXT1PJMYj6ua\nPsl8p54JghWwt4bcaiBoB4EqmMFM9w1Do57n4++tVm+oOcdZIoL+4tlzFF86fKeD\nVtTI6zYCZ/UylYMWSAYoEGu5d2ZJPzBkkfOtmt5ljVG1ojtryT6Fh9sh4YHaZbvp\nM73CifiXkHJNsPU7tmx6MYVc/tvc/KeyMknmgxUKu/wvo3Xw8VRfZ/d2oV1eBSiS\n+FNB4YMrKNn3+n5inocBUrD2F6xCeMam6RsjQnhK9F3eW3Q+WYG2rqO8pMOV4568\nKnIh6HxEdjxpFMnJL/dxG0qFbwvUmpnS4VEFcOevOay4hcF5jPT6SR3yhWGqz5X/\nF4CoVp8TZcVNo48/2eCicnRasnW3CEsYSQgF0J02WfmrIjcN3go9xWNibgLazLqB\nHWqtHTRtv5WxY/dui2KpNUekVW4g5rtQyL5lnjgqyXYR9RmeQxdQcj/WR3R6WMAY\nsmNNZ9fpXhFs8qbtQ193lpamHhr+51MlVJ96SZRTgV8CPJdMcroQHvOU9h9+TNMq\ntkhaAJ3Iq7HC8ysY0IAUivC6giL3UIQmyjTDOz5Drpu2bYGoVohOFf/4dQARAQAB\nzRBQb29yTWFuSGFuZHNoYWtlwsGGBBMBCgAwBQJlt8hEAhsGAwsJDQMVCggEFgMB\nAAIeARYhBBSbO6xQpD2YnD2731M5OwEyphEGAAoJEFM5OwEyphEGgloP/jgth0j0\n+SaE2Ag0PlPtCf+oRV1bbm3x/tBYmz1C2+PrKA4pULX/UYH2i+cvCpan56KcMO7f\nv54AsEu60QDg61dqAm0xxv2ow+qTbioEBJChOyKbxp4tof2Q1H3iJ8hzAqTuSuJH\nPCJ69vy1qGnvz+JHpgXERLJtTWo5TED7zP6Qz2qT/ErnJz0sft8J3+QfgyfSiXa+\nHq/sZuNh93Vo8hmtrvevAdfkYKrsQb9XmQljpVjrr0BnYaslRMn7xEm7ZX3UB3g0\nlGtyz/jnRfj1bT7/UjK6OV4HKAxZFg/JrTjpePYDaHmx4BqelUVi97LZdlrqYbuH\nzefq/qRcxxv3Iw0tEv7W3fT0rTu/JW19XDkD921C9fQ+E64v1M+NYDnQEj5Mv8/w\n9fRuhXrqBRs92ubjErgrdxiN/Pucml9sMLk7nq+lnuG0DIkiYENUmmZ17yeIRN5A\n6FuQIKm+GstvEoXVkSdw3xA7qnU34J6LM82dXRd5l8WHoC+ENnd+A60MmyiOArrB\nDREG8uXU/jq7knUE6AQTrJunY3ZjEC1RRS74k6kjp+Ba9QXXwL0y9axQLS4D8Fju\np+mw5GkYRRcbec6eW8wPjmrfxmq/6QwuqDaQ8q0/et9KIJ+URGO5XioC9AtUeW9c\nTDA/dahm/ErH6kqXsMj4XtYc0iyGJ4IZ/MmF\n=dChJ\n-----END PGP PUBLIC KEY BLOCK-----\n', 'peer': 'VoiceSatelliteV0.3.0:10.17.2.127::3d844abe-02a0-4c63-b908-29097adfabb7', 'node_id': 'master:0.0.0.0', 'session_id': '3d844abe-02a0-4c63-b908-29097adfabb7'}
2024-02-01 11:48:10.133 - HiveMind-voice-sat - hivemind_bus_client.protocol:handle_hello:137 - DEBUG - session_id updated to: 3d844abe-02a0-4c63-b908-29097adfabb7
2024-02-01 11:48:10.143 - HiveMind-voice-sat - hivemind_bus_client.protocol:handle_handshake:175 - INFO - HANDSHAKE: {'handshake': False, 'min_protocol_version': 0, 'max_protocol_version': 1, 'binarize': True, 'preshared_key': True, 'password': True, 'crypto_required': True}
2024-02-01 11:48:10.154 - HiveMind-voice-sat - hivemind_bus_client.protocol:start_handshake:141 - INFO - hivemind supports binarization protocol
2024-02-01 11:48:10.165 - HiveMind-voice-sat - hivemind_bus_client.client:emit:279 - DEBUG - sending to HiveMind: HiveMessageType.HANDSHAKE
2024-02-01 11:48:10.180 - HiveMind-voice-sat - ovos_bus_client.client.client:on_error:116 - WARNING - Could not send message because connection has closed
2024-02-01 11:48:10.187 - HiveMind-voice-sat - ovos_bus_client.client.client:on_error:127 - ERROR - Exception closing websocket at ws://10.17.3.102:5678?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk=: Connection to remote host was lost.
2024-02-01 11:48:10.198 - HiveMind-voice-sat - ovos_bus_client.client.client:on_error:129 - WARNING - Message Bus Client will reconnect in 5.0 seconds.
2024-02-01 11:48:15.237 - HiveMind-voice-sat - ovos_bus_client.client.client:on_open:93 - DEBUG - Connected
2024-02-01 11:48:15.251 - HiveMind-voice-sat - hivemind_bus_client.client:emit:279 - DEBUG - sending to HiveMind: HiveMessageType.BUS
2024-02-01 11:48:15.391 - HiveMind-voice-sat - hivemind_bus_client.protocol:handle_hello:129 - INFO - HELLO: {'pubkey': '-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nxsFNBGW3yEIBEAC5n/NGeSNmTOlSBuZS5U3+0HH8wGTeqsXSuMGv+1XWjbkI5ALU\nSyNMrZBUhtvDtDc1QYx5i0qA26anUbCCOVuRvxFOT1IzjLhAg1hRGXT1PJMYj6ua\nPsl8p54JghWwt4bcaiBoB4EqmMFM9w1Do57n4++tVm+oOcdZIoL+4tlzFF86fKeD\nVtTI6zYCZ/UylYMWSAYoEGu5d2ZJPzBkkfOtmt5ljVG1ojtryT6Fh9sh4YHaZbvp\nM73CifiXkHJNsPU7tmx6MYVc/tvc/KeyMknmgxUKu/wvo3Xw8VRfZ/d2oV1eBSiS\n+FNB4YMrKNn3+n5inocBUrD2F6xCeMam6RsjQnhK9F3eW3Q+WYG2rqO8pMOV4568\nKnIh6HxEdjxpFMnJL/dxG0qFbwvUmpnS4VEFcOevOay4hcF5jPT6SR3yhWGqz5X/\nF4CoVp8TZcVNo48/2eCicnRasnW3CEsYSQgF0J02WfmrIjcN3go9xWNibgLazLqB\nHWqtHTRtv5WxY/dui2KpNUekVW4g5rtQyL5lnjgqyXYR9RmeQxdQcj/WR3R6WMAY\nsmNNZ9fpXhFs8qbtQ193lpamHhr+51MlVJ96SZRTgV8CPJdMcroQHvOU9h9+TNMq\ntkhaAJ3Iq7HC8ysY0IAUivC6giL3UIQmyjTDOz5Drpu2bYGoVohOFf/4dQARAQAB\nzRBQb29yTWFuSGFuZHNoYWtlwsGGBBMBCgAwBQJlt8hEAhsGAwsJDQMVCggEFgMB\nAAIeARYhBBSbO6xQpD2YnD2731M5OwEyphEGAAoJEFM5OwEyphEGgloP/jgth0j0\n+SaE2Ag0PlPtCf+oRV1bbm3x/tBYmz1C2+PrKA4pULX/UYH2i+cvCpan56KcMO7f\nv54AsEu60QDg61dqAm0xxv2ow+qTbioEBJChOyKbxp4tof2Q1H3iJ8hzAqTuSuJH\nPCJ69vy1qGnvz+JHpgXERLJtTWo5TED7zP6Qz2qT/ErnJz0sft8J3+QfgyfSiXa+\nHq/sZuNh93Vo8hmtrvevAdfkYKrsQb9XmQljpVjrr0BnYaslRMn7xEm7ZX3UB3g0\nlGtyz/jnRfj1bT7/UjK6OV4HKAxZFg/JrTjpePYDaHmx4BqelUVi97LZdlrqYbuH\nzefq/qRcxxv3Iw0tEv7W3fT0rTu/JW19XDkD921C9fQ+E64v1M+NYDnQEj5Mv8/w\n9fRuhXrqBRs92ubjErgrdxiN/Pucml9sMLk7nq+lnuG0DIkiYENUmmZ17yeIRN5A\n6FuQIKm+GstvEoXVkSdw3xA7qnU34J6LM82dXRd5l8WHoC+ENnd+A60MmyiOArrB\nDREG8uXU/jq7knUE6AQTrJunY3ZjEC1RRS74k6kjp+Ba9QXXwL0y9axQLS4D8Fju\np+mw5GkYRRcbec6eW8wPjmrfxmq/6QwuqDaQ8q0/et9KIJ+URGO5XioC9AtUeW9c\nTDA/dahm/ErH6kqXsMj4XtYc0iyGJ4IZ/MmF\n=dChJ\n-----END PGP PUBLIC KEY BLOCK-----\n', 'peer': 'VoiceSatelliteV0.3.0:10.17.2.127::42060477-923a-4c94-af33-53ecff8e9e07', 'node_id': 'master:0.0.0.0', 'session_id': '42060477-923a-4c94-af33-53ecff8e9e07'}
2024-02-01 11:48:15.403 - HiveMind-voice-sat - hivemind_bus_client.protocol:handle_hello:137 - DEBUG - session_id updated to: 42060477-923a-4c94-af33-53ecff8e9e07
2024-02-01 11:48:15.417 - HiveMind-voice-sat - hivemind_bus_client.protocol:handle_handshake:175 - INFO - HANDSHAKE: {'handshake': False, 'min_protocol_version': 0, 'max_protocol_version': 1, 'binarize': True, 'preshared_key': True, 'password': True, 'crypto_required': True}
2024-02-01 11:48:15.429 - HiveMind-voice-sat - hivemind_bus_client.protocol:start_handshake:141 - INFO - hivemind supports binarization protocol
2024-02-01 11:48:15.443 - HiveMind-voice-sat - hivemind_bus_client.client:emit:279 - DEBUG - sending to HiveMind: HiveMessageType.HANDSHAKE
2024-02-01 11:48:15.455 - HiveMind-voice-sat - ovos_bus_client.client.client:on_error:116 - WARNING - Could not send message because connection has closed
2024-02-01 11:48:15.466 - HiveMind-voice-sat - ovos_bus_client.client.client:on_error:127 - ERROR - Exception closing websocket at ws://10.17.3.102:5678?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk=: Connection to remote host was lost.
2024-02-01 11:48:15.475 - HiveMind-voice-sat - ovos_bus_client.client.client:on_error:129 - WARNING - Message Bus Client will reconnect in 5.0 seconds.

Listener log:

hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:28.584 - HiveMind - hivemind_core.service:on_close:176 - INFO - disconnecting client: VoiceSatelliteV0.3.0:10.17.2.127::f1575d64-a208-4625-9195-eb445d2c0340
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:33.657 - HiveMind - hivemind_core.service:open:124 - INFO - authorizing client: VoiceSatelliteV0.3.0
hivemind-listener-745d99c479-q4ssv hivemind-listener Uncaught exception GET /?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk= (10.17.2.127)
hivemind-listener-745d99c479-q4ssv hivemind-listener HTTPServerRequest(protocol='http', host='10.17.3.102:5678', method='GET', uri='/?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk=', version='HTTP/1.1', remote_ip='10.17.2.127')
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 154, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     return cipher.decrypt_and_verify(ciphertext, tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 567, in decrypt_and_verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     self.verify(received_mac_tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 508, in verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise ValueError("MAC check failed")
hivemind-listener-745d99c479-q4ssv hivemind-listener ValueError: MAC check failed
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener During handling of the above exception, another exception occurred:
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/tornado/websocket.py", line 631, in _run_callback
hivemind-listener-745d99c479-q4ssv hivemind-listener     result = callback(*args, **kwargs)
hivemind-listener-745d99c479-q4ssv hivemind-listener              ^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/service.py", line 111, in on_message
hivemind-listener-745d99c479-q4ssv hivemind-listener     message = self.client.decode(message)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/protocol.py", line 121, in decode
hivemind-listener-745d99c479-q4ssv hivemind-listener     payload = decrypt_bin(self.crypto_key, payload)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 156, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener hivemind_bus_client.exceptions.DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:33.841 - HiveMind - hivemind_core.service:on_close:176 - INFO - disconnecting client: VoiceSatelliteV0.3.0:10.17.2.127::3d844abe-02a0-4c63-b908-29097adfabb7
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:38.890 - HiveMind - hivemind_core.service:open:124 - INFO - authorizing client: VoiceSatelliteV0.3.0
hivemind-listener-745d99c479-q4ssv hivemind-listener Uncaught exception GET /?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk= (10.17.2.127)
hivemind-listener-745d99c479-q4ssv hivemind-listener HTTPServerRequest(protocol='http', host='10.17.3.102:5678', method='GET', uri='/?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk=', version='HTTP/1.1', remote_ip='10.17.2.127')
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 154, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     return cipher.decrypt_and_verify(ciphertext, tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 567, in decrypt_and_verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     self.verify(received_mac_tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 508, in verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise ValueError("MAC check failed")
hivemind-listener-745d99c479-q4ssv hivemind-listener ValueError: MAC check failed
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener During handling of the above exception, another exception occurred:
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/tornado/websocket.py", line 631, in _run_callback
hivemind-listener-745d99c479-q4ssv hivemind-listener     result = callback(*args, **kwargs)
hivemind-listener-745d99c479-q4ssv hivemind-listener              ^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/service.py", line 111, in on_message
hivemind-listener-745d99c479-q4ssv hivemind-listener     message = self.client.decode(message)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/protocol.py", line 121, in decode
hivemind-listener-745d99c479-q4ssv hivemind-listener     payload = decrypt_bin(self.crypto_key, payload)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 156, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener hivemind_bus_client.exceptions.DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:39.123 - HiveMind - hivemind_core.service:on_close:176 - INFO - disconnecting client: VoiceSatelliteV0.3.0:10.17.2.127::42060477-923a-4c94-af33-53ecff8e9e07
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:44.165 - HiveMind - hivemind_core.service:open:124 - INFO - authorizing client: VoiceSatelliteV0.3.0
hivemind-listener-745d99c479-q4ssv hivemind-listener Uncaught exception GET /?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk= (10.17.2.127)
hivemind-listener-745d99c479-q4ssv hivemind-listener HTTPServerRequest(protocol='http', host='10.17.3.102:5678', method='GET', uri='/?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk=', version='HTTP/1.1', remote_ip='10.17.2.127')
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 154, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     return cipher.decrypt_and_verify(ciphertext, tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 567, in decrypt_and_verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     self.verify(received_mac_tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 508, in verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise ValueError("MAC check failed")
hivemind-listener-745d99c479-q4ssv hivemind-listener ValueError: MAC check failed
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener During handling of the above exception, another exception occurred:
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/tornado/websocket.py", line 631, in _run_callback
hivemind-listener-745d99c479-q4ssv hivemind-listener     result = callback(*args, **kwargs)
hivemind-listener-745d99c479-q4ssv hivemind-listener              ^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/service.py", line 111, in on_message
hivemind-listener-745d99c479-q4ssv hivemind-listener     message = self.client.decode(message)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/protocol.py", line 121, in decode
hivemind-listener-745d99c479-q4ssv hivemind-listener     payload = decrypt_bin(self.crypto_key, payload)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 156, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener hivemind_bus_client.exceptions.DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:44.345 - HiveMind - hivemind_core.service:on_close:176 - INFO - disconnecting client: VoiceSatelliteV0.3.0:10.17.2.127::3a92b534-f6b8-4d6d-a23a-53a36816d1d1
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:49.453 - HiveMind - hivemind_core.service:open:124 - INFO - authorizing client: VoiceSatelliteV0.3.0
hivemind-listener-745d99c479-q4ssv hivemind-listener Uncaught exception GET /?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk= (10.17.2.127)
hivemind-listener-745d99c479-q4ssv hivemind-listener HTTPServerRequest(protocol='http', host='10.17.3.102:5678', method='GET', uri='/?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk=', version='HTTP/1.1', remote_ip='10.17.2.127')
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 154, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     return cipher.decrypt_and_verify(ciphertext, tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 567, in decrypt_and_verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     self.verify(received_mac_tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 508, in verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise ValueError("MAC check failed")
hivemind-listener-745d99c479-q4ssv hivemind-listener ValueError: MAC check failed
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener During handling of the above exception, another exception occurred:
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/tornado/websocket.py", line 631, in _run_callback
hivemind-listener-745d99c479-q4ssv hivemind-listener     result = callback(*args, **kwargs)
hivemind-listener-745d99c479-q4ssv hivemind-listener              ^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/service.py", line 111, in on_message
hivemind-listener-745d99c479-q4ssv hivemind-listener     message = self.client.decode(message)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/protocol.py", line 121, in decode
hivemind-listener-745d99c479-q4ssv hivemind-listener     payload = decrypt_bin(self.crypto_key, payload)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 156, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener hivemind_bus_client.exceptions.DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:49.637 - HiveMind - hivemind_core.service:on_close:176 - INFO - disconnecting client: VoiceSatelliteV0.3.0:10.17.2.127::431ee671-aa7e-46a2-b9c9-54e90147da65
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:54.770 - HiveMind - hivemind_core.service:open:124 - INFO - authorizing client: VoiceSatelliteV0.3.0
hivemind-listener-745d99c479-q4ssv hivemind-listener Uncaught exception GET /?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk= (10.17.2.127)
hivemind-listener-745d99c479-q4ssv hivemind-listener HTTPServerRequest(protocol='http', host='10.17.3.102:5678', method='GET', uri='/?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk=', version='HTTP/1.1', remote_ip='10.17.2.127')
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 154, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     return cipher.decrypt_and_verify(ciphertext, tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 567, in decrypt_and_verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     self.verify(received_mac_tag)
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/Cryptodome/Cipher/_mode_gcm.py", line 508, in verify
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise ValueError("MAC check failed")
hivemind-listener-745d99c479-q4ssv hivemind-listener ValueError: MAC check failed
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener During handling of the above exception, another exception occurred:
hivemind-listener-745d99c479-q4ssv hivemind-listener 
hivemind-listener-745d99c479-q4ssv hivemind-listener Traceback (most recent call last):
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/tornado/websocket.py", line 631, in _run_callback
hivemind-listener-745d99c479-q4ssv hivemind-listener     result = callback(*args, **kwargs)
hivemind-listener-745d99c479-q4ssv hivemind-listener              ^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/service.py", line 111, in on_message
hivemind-listener-745d99c479-q4ssv hivemind-listener     message = self.client.decode(message)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_core/protocol.py", line 121, in decode
hivemind-listener-745d99c479-q4ssv hivemind-listener     payload = decrypt_bin(self.crypto_key, payload)
hivemind-listener-745d99c479-q4ssv hivemind-listener               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hivemind-listener-745d99c479-q4ssv hivemind-listener   File "/home/hivemind/.venv/lib/python3.11/site-packages/hivemind_bus_client/util.py", line 156, in decrypt_bin
hivemind-listener-745d99c479-q4ssv hivemind-listener     raise DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener hivemind_bus_client.exceptions.DecryptionKeyError
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:47:54.953 - HiveMind - hivemind_core.service:on_close:176 - INFO - disconnecting client: VoiceSatelliteV0.3.0:10.17.2.127::a37d02a6-4e59-449c-a0c6-a80c53aa16a8
hivemind-listener-745d99c479-q4ssv hivemind-listener 2024-02-01 16:48:00.106 - HiveMind - hivemind_core.service:open:124 - INFO - authorizing client: VoiceSatelliteV0.3.0
hivemind-listener-745d99c479-q4ssv hivemind-listener Uncaught exception GET /?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk= (10.17.2.127)
hivemind-listener-745d99c479-q4ssv hivemind-listener HTTPServerRequest(protocol='http', host='10.17.3.102:5678', method='GET', uri='/?authorization=Vm9pY2VTYXRlbGxpdGVWMC4zLjA6ODBmN2U1YTRlNWIwYzg3ZDY3M2VmM2M5NzE2ODA0NTk=', version='HTTP/1.1', remote_ip='10.17.2.127')
JarbasAl commented 6 months ago

i didnt have time to dig deeper into this one yet, but i think its just the key rotation getting out of sync, on reconnection it tries to use the old key instead of doing the new handshake properly

sometimes it will work, others not, guess it depends on what message gets sent first, some weird race condition

goldyfruit commented 1 month ago

Adding some more logs.

Jun 29 08:49:05 x270.home.lan hivemind-voice-sat[2054]: 2024-06-29 08:49:05.583 - HiveMind-voice-sat - ovos_dinkum_listener.voice_loop.voice_loop:run:234 - INFO - Wakeword detected
Jun 29 08:49:08 x270.home.lan hivemind-voice-sat[2054]: 2024-06-29 08:49:08.212 - HiveMind-voice-sat - ovos_dinkum_listener.voice_loop.voice_loop:run:268 - INFO - speech finished
Jun 29 08:49:08 x270.home.lan hivemind-voice-sat[2054]: 2024-06-29 08:49:08.860 - HiveMind-voice-sat - ovos_dinkum_listener.voice_loop.voice_loop:_after_cmd:778 - INFO - transcribed: [('what are you', 0.96850061)]
Jun 29 08:49:08 x270.home.lan hivemind-voice-sat[2054]: 2024-06-29 08:49:08.864 - HiveMind-voice-sat - hivemind_bus_client.client:emit:333 - WARNING - Could not send HiveMessageType.BUS message because connection has been closed
Jun 29 08:49:08 x270.home.lan hivemind-voice-sat[2054]: 2024-06-29 08:49:08.867 - HiveMind-voice-sat - hivemind_bus_client.client:emit:333 - WARNING - Could not send HiveMessageType.BUS message because connection has been closed