local broker

[Anto79-ops]

Hi, I'd like to get this to work using a local broker, a Ubuntu machine on the same network as my RPi Zero. Running your script on ths RPi and then I changed the broker address to my Ubuntu machine address,

[mqtt]
broker = 198.168.0.68     # Address to the mqtt broker
prefix = solar                # All topics are prefixed by "prefix/"
                              # E.g. prefix/battery_1/voltage

it seems its is not accepting a valid IP address but interpreted as a hostname. Is there an option for this to accecpt a local broker with an ip address?

pi@raspberrypi:~/solar-monitor $ python3 solar-monitor.py
Traceback (most recent call last):
  File "/home/pi/solar-monitor/solar-monitor.py", line 48, in <module>
    datalogger = DataLogger(config)
  File "/home/pi/solar-monitor/datalogger.py", line 184, in __init__
    self.mqtt = DataLoggerMqtt(config.get('mqtt', 'broker'), 1883, prefix=config.get('mqtt', 'prefix'), username=config.get('mqtt', 'username'), password=config.get('mqtt', 'password'), hostname=config.get('mqtt', 'hostname'))
  File "/home/pi/solar-monitor/datalogger.py", line 29, in __init__
    self.client.connect(broker, port)                                   # establish connection
  File "/home/pi/.local/lib/python3.9/site-packages/paho/mqtt/client.py", line 914, in connect
    return self.reconnect()
  File "/home/pi/.local/lib/python3.9/site-packages/paho/mqtt/client.py", line 1044, in reconnect
    sock = self._create_socket_connection()
  File "/home/pi/.local/lib/python3.9/site-packages/paho/mqtt/client.py", line 3685, in _create_socket_connection
    return socket.create_connection(addr, timeout=self._connect_timeout, source_address=source)
  File "/usr/lib/python3.9/socket.py", line 822, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

[Olen]

This is really strange. I simply call client.connect(broker, port) (port seems to be hardcoded to 1833, I should probably fix that too some time) on the Paho Client object, and the documentation states that

The connect() function connects the client to a broker. This is a blocking function. It takes the following arguments:

host the hostname or IP address of the remote broker

So nothing in my code makes any distinction between an IP-address and a hostname.
You could insert a simple

        logging.debug("Connecting to {}".format(broker))

Just before the connection is done (around line 28 of datalogger.py), just to make sure the IP-address is correctly fetched from the config. You would also need to set debug = True in the [monitor]-section in solar-monitor.ini

Also, are you able to connect to the broker using e.g. telnet?

$ telnet 198.168.0.68 1883

[hardillb]

Without digging into what's reading the config this looks wrong hostname=config.get('mqtt', 'hostname')

It looks to be looking for the key hostname not broker under the mqtt block and trying to connect to that

        if not hostname:
            hostname = socket.gethostname()
        self.client = paho.Client("{}".format(hostname))

Should be

        if not hostname:
            hostname = socket.gethostname()
        self.client = paho.Client("{}".format(broker))

[Olen]

"hostname" in this setting is the local hostname of the monitor-device. It should probably be renamed to "client_id" or something to match the Paho variable names.

I use the hostname to differenciate between the clients, hence the variable name just stuck:

        if not hostname:
            hostname = socket.gethostname()

This code needs a lot of cleanup...

[Anto79-ops]

thanks, all for your input so far. To answer some of your question/suggestions here's an update

1. Yes, I can telnet sucesfully from the RPi Zero to the Ubuntu machine via $ telnet 192.168.1.86 1883

2. Changing the 'hostname' to 'broker' here self.client = paho.Client("{}".format(broker))

with debugging on and a logging dubug rule, results in this error:

pi@raspberrypi:~/solar-monitor $ python3 solar-monitor.py
Debug enabled
DEBUG: Creating new DataLogger
DEBUG: Creating new MQTT-logger
DEBUG: Connecting to 192.168.1.86     # Address to the mqtt broker
Traceback (most recent call last):
  File "/home/pi/solar-monitor/solar-monitor.py", line 48, in <module>
    datalogger = DataLogger(config)
  File "/home/pi/solar-monitor/datalogger.py", line 185, in __init__
    self.mqtt = DataLoggerMqtt(config.get('mqtt', 'broker'), 1883, prefix=config.get('mqtt', 'prefix'), username=config.get('mqtt', 'username'), password=config.get('mqtt', 'password'), hostname=config.get('mqtt', 'hostname'))
  File "/home/pi/solar-monitor/datalogger.py", line 30, in __init__
    self.client.connect(broker, port)                                   # establish connection
  File "/home/pi/.local/lib/python3.9/site-packages/paho/mqtt/client.py", line 914, in connect
    return self.reconnect()
  File "/home/pi/.local/lib/python3.9/site-packages/paho/mqtt/client.py", line 1044, in reconnect
    sock = self._create_socket_connection()
  File "/home/pi/.local/lib/python3.9/site-packages/paho/mqtt/client.py", line 3685, in _create_socket_connection
    return socket.create_connection(addr, timeout=self._connect_timeout, source_address=source)
  File "/usr/lib/python3.9/socket.py", line 822, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

2. Changing the 'hostname' to 'client_id' results in this error

pi@raspberrypi:~/solar-monitor $ python3 solar-monitor.py
Debug enabled
DEBUG: Creating new DataLogger
DEBUG: Creating new MQTT-logger
Traceback (most recent call last):
  File "/home/pi/solar-monitor/solar-monitor.py", line 48, in <module>
    datalogger = DataLogger(config)
  File "/home/pi/solar-monitor/datalogger.py", line 185, in __init__
    self.mqtt = DataLoggerMqtt(config.get('mqtt', 'broker'), 1883, prefix=config.get('mqtt', 'prefix'), username=config.get('mqtt', 'username'), password=config.get('mqtt', 'password'), hostname=config.get('mqtt', 'hostname'))
  File "/home/pi/solar-monitor/datalogger.py", line 20, in __init__
    self.client = paho.Client("{}".format(client_id))        #  create client object
NameError: name 'client_id' is not defined

The debugger says its connecting, right? so maybe its not a connection issue afterall?

by the way, I;ve put in empty values for username, password, hostname in the ini file otherwise it would complain about not having these values....

[mqtt]
broker = 192.168.1.86     # Address to the mqtt broker
prefix = solar            # All topics are prefixed by "prefix/" # E.g. prefix/battery_1/voltage
username =    
password = 
hostname = 

[Olen]

Just to be clear. The change to "client_id" needs to be reflected multiple places, so it is not a simple matter of changing it there.

And "broker" and "hostname" are two different things. "broker" is the hostname or address of the MQTT-broker. "hostname" is the hostname (or whatever you want to use) of the client that this code is running at, used to distinguish this mqtt-client from other clients connecting to the same broker.

The last variable - hostname - is probably an ambigious name, and should be change throghout the code to e.g. "_clientid". But you can't do that just one place.

Now, why you are unable to connect to the broker is not easy to tell.

You could try with some minimal code in a mqtt-test.py or something:

import paho.mqtt.client as paho
client = paho.Client("testclient")
client.connect("192.168.1.86", 1833)

I think that should be enough to get you connected.

[Anto79-ops]

HA thanks!

I simply replaced line code

self.client.connect(broker, port) # establish connection

with this

self.client.connect("192.168.1.86", 1883) # establish connection

and its working...or at least its trying to connect to the BT-1

[Anto79-ops]

Ok, its working well now, its collecting and sending data to the broker. I plan on using Grafana on the other end.

May I ask, how can you increase the number of signficant digits? for example, amps, could it show something like 0.05 amps, rather than 0.1 amps? and for other values also, could be good to have 2 decimal places.

[Olen]

Glad to hear. Still not sure why there is a difference between using the variable and just hardcoding the IP-address.

When it comes to number of digits, I don't think I want to do that. At least here, the fluctations up and down are so rapid that the number of mqtt-requests simply were overflowing the system. Also, I am running this in remote locations with only 4G/LTE-connections, so to save a bit of data, I only send the updates when they are "large enough". I still get updates from each sensor multiple times a minute (when the sun is shining), so I don't really think it would give much extra granularity to have more decimals.

[Anto79-ops]

thanks! Yes, I don't know either, but it works for me.

regarding the significant digits, for sure, but if I wanted to change it on my local script, which string should I/can I format for that to work?

[Olen]

If you look at the class PowerDevice in solaredevice.py you can find the definitions of the various monitored values.

You will mostly see that values are stored as "high resolution" (e.g. millivolt, temperature in kelvin etc), but it will report/log on more "human friendly" values (volt, temperature in celsius etc). I also have various checks in place because quite often these devices report crazy values for a short time, so I try to filter out values that are obviously bad.

I think (but it's bee a while since I worked on this) that line 209-212 and further down in solaredevice.py contains a list of all items that should be logged remotely.

So you could change the list on line 209... so that e.g. input_voltage changed to input_mvoltage wich would get you the value in milivolts instead of volts. You could then do any roundings and calculations in the receiving end and use as many decimals as you need. And if you want fahrenheit instead of celsius, you could change line 223 from self.entities.temperature_celsius to self.entities.temperature_fahrenheit

[Anto79-ops]

thanks for the above. I think actually, its good as is like you said.

I'm curious about the datalogging aspect of this now. I'm using Grafana MQTT https://grafana.com/blog/2021/08/12/streaming-real-time-sensor-data-to-grafana-using-mqtt-and-grafana-live/, I noticed when put my broker listen to all messages from the broker by using this command

mosquitto_sub -v -h broker_ip -p 1883 -t '#'

i get all this:

homeassistant/sensor/regulator/charge_current/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Charge Current", "ud": "solar            # All topics are prefixed by \"prefix/\"_regulator_charge_current", "state_topic": "solar            # All topics are prefixed bfix/\"/regulator/charge_current/state", "force_update": true, "icon": "mdi:current-dc", "unit_of_measurement": "A"}
homeassistant/sensor/regulator/voltage/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Voltage", "unique_id": "so        # All topics are prefixed by \"prefix/\"_regulator_voltage", "state_topic": "solar            # All topics are prefixed by \"prefix/\"/regulatage/state", "force_update": true, "icon": "mdi:flash", "unit_of_measurement": "V"}
homeassistant/sensor/regulator/input_voltage/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Input Voltage", "uni: "solar            # All topics are prefixed by \"prefix/\"_regulator_input_voltage", "state_topic": "solar            # All topics are prefixed by \/\"/regulator/input_voltage/state", "force_update": true, "icon": "mdi:flash", "unit_of_measurement": "V"}
homeassistant/sensor/regulator/charge_voltage/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Charge Voltage", "ud": "solar            # All topics are prefixed by \"prefix/\"_regulator_charge_voltage", "state_topic": "solar            # All topics are prefixed bfix/\"/regulator/charge_voltage/state", "force_update": true, "icon": "mdi:flash", "unit_of_measurement": "V"}
homeassistant/sensor/regulator/power/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Power", "unique_id": "solar     # All topics are prefixed by \"prefix/\"_regulator_power", "state_topic": "solar            # All topics are prefixed by \"prefix/\"/regulator/powe", "force_update": true, "device_class": "power", "unit_of_measurement": "W"}
homeassistant/sensor/regulator/input_power/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Input Power", "unique_olar            # All topics are prefixed by \"prefix/\"_regulator_input_power", "state_topic": "solar            # All topics are prefixed by \"prefigulator/input_power/state", "force_update": true, "device_class": "power", "unit_of_measurement": "W"}
homeassistant/sensor/regulator/charge_power/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Charge Power", "uniqu"solar            # All topics are prefixed by \"prefix/\"_regulator_charge_power", "state_topic": "solar            # All topics are prefixed by \"pr/regulator/charge_power/state", "force_update": true, "device_class": "power", "unit_of_measurement": "W"}
homeassistant/sensor/regulator/soc/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Soc", "unique_id": "solar     # All topics are prefixed by \"prefix/\"_regulator_soc", "state_topic": "solar            # All topics are prefixed by \"prefix/\"/regulator/soc/statece_update": true, "device_class": "battery", "unit_of_measurement": "%"}
homeassistant/sensor/regulator/capacity/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Capacity", "unique_id": "          # All topics are prefixed by \"prefix/\"_regulator_capacity", "state_topic": "solar            # All topics are prefixed by \"prefix/\"/reguapacity/state", "force_update": true, "icon": "mdi:solar-power"}
homeassistant/sensor/regulator/temperature/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Temperature", "unique_olar            # All topics are prefixed by \"prefix/\"_regulator_temperature", "state_topic": "solar            # All topics are prefixed by \"prefigulator/temperature/state", "force_update": true, "device_class": "temperature", "unit_of_measurement": "\u00b0C"}
homeassistant/switch/regulator/power_switch/config {"name": "Solar            # all topics are prefixed by \"prefix/\" Regulator Power Switch", "uniqu"solar            # All topics are prefixed by \"prefix/\"_regulator_power_switch", "state_topic": "solar            # All topics are prefixed by \"pr/regulator/power_switch/state", "command_topic": "solar            # All topics are prefixed by \"prefix/\"/regulator/power_switch/set", "payload_on":yload_off": 0}

I'm just wondering if this is some configuration issue, and whether its sending it out for Home Assistant. How do I send it out as regular MQTT so that it can work with my Grafana MQTT.

[Olen]

You should be able to use it fine. You can ignore the homeassistant-topic. That is just config to automatically add the entities to HA. The topic you should listen to is whatever you set as "prefix" in your .ini-file (or "solar-monitor" if you have not set a specifik prefix).

However, if that is what you receive, I think I understand why you were unable to connect. Comments in the ini-file are not supported, so you need to remove all the comments (and #-es) from the .ini-file.

[Anto79-ops]

yes....thank you! Grafana MQTT is now reading the broker output! Cheers

[Anto79-ops]

Here's the final product, using InfluxDB in Grafana (on Ubuntu) and Red-NODE (Raspberry Pi Zero 2 W) collecting and logging the data from my BT-1 module of my Renogy Rover 40A charge contoller! All local operation.

https://youtu.be/BKZpb4P_6eY

thanks for sharing your script

[Olen]

I am glad you got it working.

I'll update the README with some of the stuff from this issue.

[Anto79-ops]

quick question if you put "True" in the reconnect part of the ini file such as below (False is default):

[regulator]
type = SolarLink
mac = 11:11:11:11:11:11
reconnect = True

does it mean it will try to reconnect on its own when/if it disconnects? If yes, how often will it try?

thanks

[Olen]

        if self.auto_reconnect:
            logging.info("[{}] Reconnecting in 10 seconds".format(self.logger_name))
            time.sleep(10)
            self.connect()

Every 10 second, it seems.