diyhue / diyHue

Main diyHue software repo
https://diyhue.org/
Other
1.56k stars 279 forks source link

MQTT with TLS #892

Open Timmmy001 opened 1 year ago

Timmmy001 commented 1 year ago

Describe the issue

I try to connect the diyhue via MQTT TLS to my FHEM Server. But I only get error messages at my FHEM server like this, which indicate that no TLS is used. On this MQTT server module is also zigbee2mqtt connected - so the mqtt instance works with TLS.

MQTT2_SERVER SSL/HTTPS error: SSL accept attempt failed error:0A00010B:SSL routines::wrong version number (peer: 192.168.188.29)

Steps you tried

Steps you tried:

  1. Add in config.yaml this parts:

    mqtt:
    mqttTls: true
    mqttTlsInsecure: true
    mqttCaCerts: /opt/hue-emulator/ca.crt

    I didn't find anything about MQTT TLS in the official doku, but this PR: https://github.com/diyhue/diyHue/pull/556

  2. restart service

  3. I get error messages in FHEM log, but no errorlogs in diyhue...

How we can help

I would like to have more information about the configuration of MQTT under diyHue.

Logs

ubuntu-desktop@Client:/opt/hue-emulator$ sudo ./HueEmulator3.py --debug
2023-02-25 18:49:11,037 - configManager.argumentHandler - INFO - Using Host 192.168.188.29:80
2023-02-25 18:49:11,042 - configManager.argumentHandler - INFO - Host MAC given as 00000000000
2023-02-25 18:49:11,042 - configManager.argumentHandler - INFO - IP range for light discovery: 0-255
2023-02-25 18:49:11,042 - configManager.argumentHandler - INFO - Deconz IP given as 127.0.0.1
2023-02-25 18:49:11,042 - configManager.argumentHandler - INFO - Online Discovery/Remote API Enabled!
2023-02-25 18:49:11,044 - configManager.argumentHandler - INFO - Using Host 192.168.188.29:80
2023-02-25 18:49:11,049 - configManager.argumentHandler - INFO - Host MAC given as 00000000000
2023-02-25 18:49:11,050 - configManager.argumentHandler - INFO - IP range for light discovery: 0-255
2023-02-25 18:49:11,050 - configManager.argumentHandler - INFO - Deconz IP given as 127.0.0.1
2023-02-25 18:49:11,050 - configManager.argumentHandler - INFO - Online Discovery/Remote API Enabled!
2023-02-25 18:49:11,050 - configManager.argumentHandler - INFO - Debug logging enabled!
2023-02-25 17:49:11,059 - configManager.configHandler - INFO - Config loaded
2023-02-25 17:49:11,120 - functions.daylightSensor - DEBUG - Daylight Sensor: location is not configured
2023-02-25 17:49:11,121 - services.mqtt - INFO - Strting MQTT service...
2023-02-25 17:49:11,121 - services.remoteDiscover - INFO - Starting discovery service
2023-02-25 17:49:11,122 - services.stateFetch - INFO - start lights sync
2023-02-25 17:49:11,124 - services.ssdp - INFO - starting ssdp...
2023-02-25 17:49:11,125 - services.ssdp - INFO - start ssdp broadcast
2023-02-25 17:49:11,125 - services.mdns - INFO - <MDNS> listener started
2023-02-25 17:49:11,126 - services.eventStreamer - DEBUG - {'creationtime': '2023-02-25T17:49:11Z', 'data': [{'children': [], 'grouped_services': [{'rid': 'f77838e4-b29c-4611-acf8-293ea5560faf', 'rtype': 'grouped_light'}], 'services': [{'rid': 'f77838e4-b29c-4611-acf8-293ea5560faf', 'rtype': 'grouped_light'}], 'id': '5f39a806-a91b-58a8-a6af-d3995ac7a4e1', 'id_v1': '/groups/0', 'metadata': {'archetype': 'other', 'name': 'Group 0'}, 'type': 'zone'}], 'id': 'd8a8eec0-7102-4069-b4e1-51badc09e95e', 'type': 'add'}
 * Serving Flask app 'HueEmulator3' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2023-02-25 17:49:11,128 - werkzeug - WARNING -  * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
2023-02-25 17:49:11,128 - werkzeug - INFO -  * Running on http://192.168.188.29:80/ (Press CTRL+C to quit)
 * Serving Flask app 'HueEmulator3' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2023-02-25 17:49:11,130 - werkzeug - WARNING -  * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
2023-02-25 17:49:11,130 - werkzeug - INFO -  * Running on https://192.168.188.29:443/ (Press CTRL+C to quit)
2023-02-25 17:49:11,611 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:11] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:49:16,644 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:16] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:49:21,682 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:21] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:49:23,136 - services.stateFetch - INFO - start lights sync
2023-02-25 17:49:26,734 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:26] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:49:31,744 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:31] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:49:36,787 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:36] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:49:38,152 - services.stateFetch - INFO - start lights sync
2023-02-25 17:49:41,814 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:41] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:49:46,832 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:46] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:49:51,876 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:51] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:49:53,168 - services.stateFetch - INFO - start lights sync
2023-02-25 17:49:56,896 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:49:56] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:01,938 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:01] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:06,961 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:06] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:08,184 - services.stateFetch - INFO - start lights sync
2023-02-25 17:50:11,991 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:11] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:16,992 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:16] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:20,425 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:20] "GET /api/45535ea8b52a11edacfc29fc2f611099/config/mqtt HTTP/1.1" 200 -
2023-02-25 17:50:21,997 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:21] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:22,193 - services.stateFetch - INFO - start lights sync
2023-02-25 17:50:25,495 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:25] "GET /static/js/540.f42e1a3d.chunk.js HTTP/1.1" 200 -
2023-02-25 17:50:25,507 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:25] "GET /api/45535ea8b52a11edacfc29fc2f611099/config/homeassistant HTTP/1.1" 200 -
2023-02-25 17:50:26,629 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:26] "GET /api/45535ea8b52a11edacfc29fc2f611099/config/mqtt HTTP/1.1" 200 -
2023-02-25 17:50:27,004 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:27] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:32,051 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:32] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:33,204 - services.stateFetch - INFO - start lights sync
2023-02-25 17:50:37,102 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:37] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:42,141 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:42] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:44,215 - services.stateFetch - INFO - start lights sync
2023-02-25 17:50:47,177 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:47] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:52,226 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:52] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:57,275 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:50:57] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:50:59,231 - services.stateFetch - INFO - start lights sync
2023-02-25 17:51:02,307 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:02] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:51:07,352 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:07] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:51:12,384 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:12] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:51:14,248 - services.stateFetch - INFO - start lights sync
2023-02-25 17:51:17,418 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:17] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:51:22,449 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:22] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:51:27,476 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:27] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:51:29,264 - services.stateFetch - INFO - start lights sync
2023-02-25 17:51:32,524 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:32] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:51:37,577 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:37] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:51:42,605 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:42] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
2023-02-25 17:51:44,280 - services.stateFetch - INFO - start lights sync
2023-02-25 17:51:47,648 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:47] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -
^C2023-02-25 17:51:52,663 - werkzeug - INFO - 192.168.188.44 - - [25/Feb/2023 17:51:52] "GET /api/45535ea8b52a11edacfc29fc2f611099/groups/0 HTTP/1.1" 200 -

Docker Info (please complete the following information):

Checklist

Additional context

mariusmotea commented 1 year ago

Give me few days, i think i miss to implement this after code refactor.

mariusmotea commented 1 year ago

Ok, i made it now. Can you check if it is working?

Timmmy001 commented 1 year ago

Thank you for the quick response! Unfortunately a partially advanced:

In Debug Log I see now a

2023-02-26 11:19:20,875 - services.remoteDiscover - INFO - Starting discovery service
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2023-02-26 11:19:20,877 - services.stateFetch - INFO - start lights sync
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
2023-02-26 11:19:20,879 - services.ssdp - INFO - start ssdp broadcast
    self._target(*self._args, **self._kwargs)
  File "/opt/hue-emulator/services/mqtt.py", line 402, in mqttServer
2023-02-26 11:19:20,879 - services.mdns - INFO - <MDNS> listener started
    mqttTlsVersion = ssl.PROTOCOL_TLS
NameError: name 'ssl' is not defined
2023-02-26 11:51:26,871 - services.mqtt - INFO - Strting MQTT service...
Exception in thread Thread-2:
2023-02-26 11:51:26,871 - services.remoteDiscover - INFO - Starting discovery service
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
2023-02-26 11:51:26,874 - services.stateFetch - INFO - start lights sync
    self._target(*self._args, **self._kwargs)
  File "/opt/hue-emulator/services/mqtt.py", line 402, in mqttServer
    mqttTlsVersion = ssl.PROTOCOL_TLS
NameError: name 'ssl' is not defined

If I start it as service (not with debug) it will be add in config.yaml:

mqtt:
 mqttCertfile: null
 mqttKeyfile: null

In the configuration of zigbee2mqtt, on the other hand, it is sufficient to store only the ca certificate, without any further cert/key file. But I don't know if this works at all with the Python implementation of mqtt in general. :-|

On FHEM MQTT side is no error log - I think there was no connection try.

mariusmotea commented 1 year ago

ssl not defined means ssl module need to be imported. I made now a new commit that import ssl module.

Timmmy001 commented 1 year ago

Next log entrys

2023-02-26 14:40:25,215 - services.remoteDiscover - INFO - Starting discovery service
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/hue-emulator/services/mqtt.py", line 404, in mqttServer
    client.tls_set(ca_certs=bridgeConfig["config"]["mqtt"]["mqttCaCerts"], certfile=bridgeConfig["config"]["mqtt"]["mqttCertfile"], keyfile=bridgeConfig["config"]["mqtt"]["mqttKeyfile"], tls_version=mqttTlsVersion)
  File "/usr/local/lib/python3.8/dist-packages/paho/mqtt/client.py", line 804, in tls_set
    context.load_verify_locations(ca_certs)
mariusmotea commented 1 year ago

in config.yaml you set mqttCaCerts to a valid location or mqttCertfile and mqttKeyfile? If not i think you should set mqttTlsInsecure to true.

Timmmy001 commented 1 year ago

That was my config part for mqtt:

mqtt:
  enabled: true
  mqttServer: SVL01.fritz.box
  mqttPort: 8883
  mqttTls: true
  mqttTlsInsecure: true
  mqttCaCerts: /opt/hue-emulator/ca.crt
  mqttUser: MQTT_TLS
  mqttPassword: xxx
  discoveryPrefix: zigbee2mqtt
  mqttCertfile: null
  mqttKeyfile: null

only ca file mqttTlsInsecure: false

File "/opt/hue-emulator/services/mqtt.py", line 404, in mqttServer
    client.tls_set(ca_certs=bridgeConfig["config"]["mqtt"]["mqttCaCerts"], certfile=bridgeConfig["config"]["mqtt"]["mqttCertfile"], keyfile=bridgeConfig["config"]["mqtt"]["mqttKeyfile"], tls_version=mqttTlsVersion)
  File "/usr/local/lib/python3.8/dist-packages/paho/mqtt/client.py", line 804, in tls_set
    context.load_verify_locations(ca_certs)
FileNotFoundError: [Errno 2] No such file or directory

only ca file mqttTlsInsecure: true

File "/opt/hue-emulator/services/mqtt.py", line 404, in mqttServer
    client.tls_set(ca_certs=bridgeConfig["config"]["mqtt"]["mqttCaCerts"], certfile=bridgeConfig["config"]["mqtt"]["mqttCertfile"], keyfile=bridgeConfig["config"]["mqtt"]["mqttKeyfile"], tls_version=mqttTlsVersion)
  File "/usr/local/lib/python3.8/dist-packages/paho/mqtt/client.py", line 804, in tls_set
    context.load_verify_locations(ca_certs)
FileNotFoundError: [Errno 2] No such file or directory

with files (cert+key) additional mqttTlsInsecure: false:

Traceback (most recent call last):
  File "/opt/hue-emulator/services/mqtt.py", line 404, in mqttServer
    client.tls_set(ca_certs=bridgeConfig["config"]["mqtt"]["mqttCaCerts"], certfile=bridgeConfig["config"]["mqtt"]["mqttCertfile"], keyfile=bridgeConfig["config"]["mqtt"]["mqttKeyfile"], tls_version=mqttTlsVersion)
  File "/usr/local/lib/python3.8/dist-packages/paho/mqtt/client.py", line 804, in tls_set

with files (cert+key) additional mqttTlsInsecure: true

Traceback (most recent call last):
  File "/opt/hue-emulator/services/mqtt.py", line 404, in mqttServer
    client.tls_set(ca_certs=bridgeConfig["config"]["mqtt"]["mqttCaCerts"], certfile=bridgeConfig["config"]["mqtt"]["mqttCertfile"], keyfile=bridgeConfig["config"]["mqtt"]["mqttKeyfile"], tls_version=mqttTlsVersion)
  File "/usr/local/lib/python3.8/dist-packages/paho/mqtt/client.py", line 804, in tls_set
    context.load_verify_locations(ca_certs)
Timmmy001 commented 1 year ago

Is there anything else I should test?

mariusmotea commented 1 year ago

I have no clue what may be wrong. I think you pasted here incomplet errors since is missing the exception on last two.