search

AnotherDaniel / smahub

SMAHub is a flexible and modular solution for collecting data from SMA photovoltaic (PV) products and publishing it to various output channels, such as MQTT.
GNU General Public License v2.0
28 stars 8 forks source link

TLS for MQTT #34

Open jardleex opened 7 months ago

jardleex commented 7 months ago

Hello,

would it be possible to support TLS on the MQTT sink? I wanted to lock down my home automation a bit more and as I got quite some stuff going on via MQTT it'll be great to secure that. For some setup's it may also be required to skip certificate validation. I'm using Lets Encrypt here which should work with enabled verification.

Thank you for your work. Do you have any way of donating to you?

AnotherDaniel commented 7 months ago

Hi @jardleex, I was afraid that someday, someone would ask for this...

Seriously though: I'm on it. Will result in a bunch of new config vars for this plugin, but that's what certificate handling gets you... ;-)

AnotherDaniel commented 7 months ago

Ok, just release v1.5.5 with TLS support for the MQTT sink plugin - give it a whirl, hope this works for you!

jardleex commented 7 months ago

Hello @AnotherDaniel

also this change works pretty good. Not the 1h time shift between smahub and iobroker comes as smahub runs on UTC time while iobroker uses TZ: Europe/Berlin. smahub seems not to use this environment variable when set. Just a minor thing.

The only thin I noticed worth mentioning is the 2024-02-07 07:45:33 ERROR Missing CA certificate error. I'm using traefik with LetsEncrypt with a valid/public domain and thus can offer valid certificates. So in this case here it's no actual error for the setup.

# docker-compose service:
  smahub:
    image: ghcr.io/anotherdaniel/smahub:v1.5.5
    restart: unless-stopped
    depends_on:
      iobroker:
        condition: service_healthy
    environment:
      SMAHUB_VERBOSE: true
      SMAHUB_VERBOSER: true
      SHM2_ENABLED: false
      TRIPOWERX_ENABLED: true
      TRIPOWERX_ADDRESS: redacted
      TRIPOWERX_USER: smahub
      TRIPOWERX_PASSWORD: 'redacted'
      TRIPOWERX_PROTOCOL: https
      TRIPOWERX_VERIFYTLS: false
      TRIPOWERX_UPDATEFREQ: 20
      MQTT_ENABLED: true
      MQTT_ADDRESS: redacted
      MQTT_PORT: 2884
      MQTT_TLS: 2
      MQTT_USER: redacted
      MQTT_PASSWORD: 'redacted'
      MQTT_UPDATEFREQ: 20
# smahub logs
2024-02-07 07:45:33 INFO     SHM2 plugin disabled
2024-02-07 07:45:33 INFO     Starting Tripower X source
2024-02-07 07:45:33 INFO     EV Charger plugin disabled
2024-02-07 07:45:33 INFO     demo source plugin disabled
2024-02-07 07:45:33 INFO     Starting MQTT sink
2024-02-07 07:45:33 INFO     HA-MQTT sink plugin disabled
2024-02-07 07:45:33 INFO     gen_ha_sensors sink plugin disabled
2024-02-07 07:45:33 ERROR    Missing CA certificate
2024-02-07 07:45:33 INFO     demo sink plugin disabled
2024-02-07 07:45:33 DEBUG    Starting new HTTPS connection (1): redacted:443
2024-02-07 07:45:33 INFO     TLS enabled
2024-02-07 07:45:33 DEBUG    MQTT sink connected to redacted:2884
2024-02-07 07:45:33 INFO     Connected to MQTT Broker
# iobroker log
mqtt.0  2024-02-07 08:45:33.400 info    Client [smahub] connected with secret 1707291933386_9697
AnotherDaniel commented 7 months ago

Have another go - timezone setting support should be there now, and that log message was a mistake.

jardleex commented 7 months ago

Confirmed for 1.5.6 that TZ environment variables works and missing CA certificate error is gone.

Thank you!

jardleex commented 7 months ago

Hi, today I've updated the MQTT instance in iobroker which caused a restart of it. smahub was not able to reconnect though, so I restarted it in the end which fixed the error message below:

2024-02-14 11:42:21 INFO     Got 401 - trying reauth
2024-02-14 12:42:28 INFO     Got 401 - trying reauth
2024-02-14 13:42:36 INFO     Got 401 - trying reauth
2024-02-14 14:42:43 INFO     Got 401 - trying reauth
2024-02-14 14:43:37 WARNING  Disconnected from MQTT Broker
Exception in thread Thread-1 (_thread_main):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 3591, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
         ^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1181, in _loop
    rc = self.loop_write()
         ^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1577, in loop_write
    rc = self._packet_write()
         ^^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 2464, in _packet_write
    write_length = self._sock_send(
                   ^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 649, in _sock_send
    return self._sock.send(buf)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1243, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2427)
2024-02-14 14:47:26 INFO     SHM2 plugin disabled
2024-02-14 14:47:26 INFO     Starting Tripower X source
2024-02-14 14:47:26 INFO     EV Charger plugin disabled
2024-02-14 14:47:26 INFO     demo source plugin disabled
2024-02-14 14:47:26 INFO     Starting MQTT sink
2024-02-14 14:47:26 INFO     HA-MQTT sink plugin disabled
2024-02-14 14:47:26 INFO     gen_ha_sensors sink plugin disabled
2024-02-14 14:47:26 INFO     demo sink plugin disabled
2024-02-14 14:47:26 INFO     TLS enabled
2024-02-14 14:47:26 INFO     Connected to MQTT Broker

Does that say something to you? Thank you!

AnotherDaniel commented 7 months ago

Does it work again after restarting all involved components, do I see that correctly? The googling I did so far is inconclusive, other than hinting at the dumpster-fire robustness that is the Python ecosystem 🫤

jardleex commented 7 months ago

Yes, everything worked again after a restart cycle. I just wanted to raise this as it might contain valuable information for fixing something. Generally the MQTT reconnect works flawless as I did multiple restarts of my broker since the new smahub feature.

I'd be okay for me to close this one here in this state.

jardleex commented 7 months ago

Nah, we can't close this one. The error comes up every time I restart the broker. The mentioned restarts must have happened when I ain't used TLS yet but after #27. I'll do some more testing and will report what I found.

AnotherDaniel commented 6 months ago

Ok, I'll also try to reproduce this.

jardleex commented 6 months ago

I did some tests again and only got the issue with MQTTS. With plain MQTT the reconnect worked just fine. I triggered the error message above by simply restarting the mqtt broker.

AnotherDaniel commented 6 months ago

@jardleex just to let you know I haven't forgotten about this issue - not that much free mind-time at the moment, and when I tried to pick this up my TLS config (that was fine the first time around) refused to work. As I said, not forgotten...

AnotherDaniel commented 6 months ago

Add-on: I just figured out what my recent issue was, and have no idea why this ever worked? Anyways - broker-restart-reconnect works for me, but that might not be related to your scenario. But please have a go with v1.6.2 and let me know how that looks for you!

jardleex commented 6 months ago

Hi, no worries, that's hobby for all of us. We as users can be happy to have able people like you to help us out of the misery the manufactures left us in.

Sadly 1.6.2 ain't changed the end result if I restart my broker:

2024-03-08 12:26:56 WARNING  Disconnected from MQTT Broker
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.name: Sunny Tripower X 25
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.configuration_url: https://REDACTED
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.identifiers: 3016101878
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.model: SMA-Sunny Tripower X 25
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.manufacturer: SMA
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.sw_version: 03.06.15.R
2024-03-08 12:26:58 DEBUG    https://REDACTED:443 "POST /api/v1/measurements/live HTTP/1.1" 200 None
Exception in thread Thread-1 (_thread_main):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
2024-03-08 12:26:58 DEBUG    value of Operation.Evt.EvtNo is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.FstStop is currently not availably (nighttime?)
    self._target(*self._args, **self._kwargs)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 3591, in _thread_main
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.HzLimRed is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.PFCtlComCfg.PF is currently not availably (nighttime?
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.PFCtlComCfg.PFExt is currently not availably (nighttime?)
    self.loop_forever(retry_first_connection=True)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.PFCtlComCfg.PFExtIn is currently not availably (nighttime?)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.PFCtlComCfg.PFIn is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.VArCtlComCfg.VArNom is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.VArCtlVolCfg.VolRef.VolRefPu is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.W is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WMaxIn is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WMaxInNomPrc is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WNom is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WSptMax is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WSptMaxNom is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WSptMin is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WSptMinNom is currently not availably (nighttime?)
    rc = self._loop(timeout)
         ^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1181, in _loop
    rc = self.loop_write()
         ^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1577, in loop_write
    rc = self._packet_write()
         ^^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 2464, in _packet_write
    write_length = self._sock_send(
                   ^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 649, in _sock_send
    return self._sock.send(buf)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1242, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2427)

Please let me know if I can provide any additional info to support you.