MQTT Test fails

[WirthmU]

I did attempt to install according to the setup directions (client version on the pi - the MQTT broker server is already running elswhere): https://github.com/markmac99/ukmon-pitools/wiki/Monitoring_with_MQTT

Installation went smoothly but upon testing I get the following error:

(vRMS) pi@raspberrypi:~/source/rms_mqtt $ python3 sendToMQTT.py test
Traceback (most recent call last):
  File "sendToMQTT.py", line 152, in <module>
    test_mqtt()
  File "sendToMQTT.py", line 138, in test_mqtt
    client = mqtt.Client(hname)
  File "/home/pi/vRMS/lib/python3.7/site-packages/paho/mqtt/client.py", line 767, in __init__
    "Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details"
ValueError: Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details

Any help to this?

[ukmda]

This is due to a breaking change in Paho MQTT 2.0 which was released a few weeks ago. See https://stackoverflow.com/questions/77984857/paho-mqtt-unsupported-callback-api-version-error

Workaround is to install version 1.6.1:

pip uninstall paho-mqtt
pip install paho-mqtt==1.6.1

[WirthmU]

Your workaround fixed the errors.

The config.ini looks like this:

(vRMS) pi@raspberrypi:~/source/rms_mqtt $ sudo cat config.ini
# config file
[mqtt]
RMSCFG=~/source/RMS/.config
BROKER=192.168.1.158
USERNAME=XXX
PASSWORD=XXX

Testing gives the following results:

(vRMS) pi@raspberrypi:~/source/rms_mqtt $ python3 sendToMQTT.py test
send to testing/raspberrypi/test, result (0, 1)

I do not know how to interpret the result (0, 1), however.

The two crontab commands themselves run without error and I did integrate it into the crontab:

(vRMS) pi@raspberrypi:~/source/rms_mqtt $ /home/pi/source/rms_mqtt/logPiStats.sh
(vRMS) pi@raspberrypi:~/source/rms_mqtt $ /home/pi/source/rms_mqtt/logMeteorStats.sh

Unfortunately nothing arrives at the broker which is located on 192.168.1.158 in the same LAN as the camera. The broker runs fine and is working with MQTT coming from another RPI camera running indi-Allsky.

Is there any way to test where the MQTT messages are going?

This are the configuration settings in indi-Allsky;

[markmac99]

Hi, the test prints out the result of a call to publish(), which is a tuple containing two values. The first value is the status - zero means success. The second is the message ID, 1 means its the first message. So, (0,1) means success.

I use MQTT Explorer to monitor my MQ server.

The messages appear under a topic "testing" and the test messages are NOT retained in MQ. What that means is, you must be connected with your monitoring client BEFORE you run the test. You should then see a new topic appear:

[WirthmU]

I did install the MQTT-explorer on the Ubuntu-PC where the Broker is running and it can effectively see the Indi-Allsky client, but not any messages from the RMS camera.

How would I need to connect the RMS client to the Broker other than providing the Broker address, credentials and running the scripts on the RMS-Pi?

[markmac99]

The status message from the test indicates that it is connecting and publishing to the target MQ broker. So the code is working as intended. If the connection had failed you would have seen a message similar to

ConnectionRefusedError: No connection could be made because the target machine actively refused it

As noted before, you must be running MQTT Explorer before you run the test, as the message is not retained on the broker. Therefore, if you run the test and then connect to look for it, you won't find it.

if you still can't see the messge you will need to look at the broker logs to debug it further.

[markmac99]

ps - double check the mq broker, username and password. I just did a quick test and if the password or username are wrong, the conenction succeeds but messages are not published.

[WirthmU]

I did as you suggested. However, still no reception of MQTT messages on the broker:

And yes the username and password have been double-checked and are correct. Is the path with just the IP-address correct? Client and broker are in the same LAN

[markmac99]

As i explained, the return code indicates that the messages are being successfully published. You will have to examine the MQ server logs to see why they're being ignored.

The reason i suggested checking username nad password is that I did a few tests, and if the password is incorrect, then the server ignores the messages.

I don't tihnk i can help you further as the code is working as expected, and it seems to be a configuration issue on the broker.

[WirthmU]

This is the error message that I get on the MQTT-server side: MQTT2_SERVER SSL/HTTPS error: SSL accept attempt failed error:0A00010B:SSL routines::wrong version number (peer: 192.168.1.61)

[markmac99]

According to Google this is an SSL configuration issue. I don't use SSL with MQ and so this code doesn't support it.

I think its probably fairly easy to do though. This article explains how to configure both the broker and the client for SSL, and in particular explains how to do it from paho-mqtt, the python lib I'm using http://www.steves-internet-guide.com/mosquitto-tls/

Because i do not use SSL, i am not calling client.tls_set() to tell the client where to find the ca.crt file and so it probably isn't offering a cert to the broker. You would need to add that. If i get time this weekend i may experiment with SSL, however i make no promises.

Meanwhile, you should also test that you can publish to your broker using the commandline mosquitto_pub client. Something like

mosquitto_pub -h yourbroker -p yourport -t test/test1 -m barfoo --cafile /path/to/ca.crt

[markmac99]

ok, i had a bit of time to look at this and have added TLS support. Tested successfully. i added the port and cafile parameters to the config file so you can provide the relevant details,, and made changes to the python code too.

[WirthmU]

It’s working now with and without TLS - great support.

Just one last question. If I run the logMeteorStats.sh it says that upload is disabled because the default station code is used (xx0001). Where do I set my own code? It would be nice if it gets it automatically or from the config.

[markmac99]

Glad its working! To get your own code, contact denis Vida denis.vida@gmail.com and proceed as explained in the installation notes for RMS in the section on Editing the Configuration File https://github.com/CroatianMeteorNetwork/RMS?tab=readme-ov-file#editing-the-configuration-file Denis will provide you with a country-specific ID and explain how to onboard to contribute data to GMN.

[WirthmU]

I guess I did explain it the wrong way. I already do have two RMS cameras up and running with codes and everything. It just seems that both bash scripts are missing my station code (CH0002). Do I have to enter it somewhere?

[markmac99]

The software reads your station code from the station's RMS .config file. The full name and path of the RMS .config file has to be put into config.ini so that it can find the details.

As for why you are seeing code XX0001: this message is coming from RMS's ConfigReader module, which i use to read the RMS .config file, and it means that ConfigReader can't find a valid code in the RMS configuration.

I suggest you check that config.ini contains the correct full path to your station's RMS .config file. By the way as i am sure you already know, during RMS's self-update, the RMS config file is temporarily replaced with a default version containing the dummy code XX0001. Once the update finishes, the original file is restored however, if you run the MQ utilities during this RMS update process, it will find the dummy code and emit the message you've seen. This is expected behaviour. Wait till RMS has finished its update process and everything should be fine.

[WirthmU]

I have problems with loading the RMS config file. The rms_mqtt config looks as follows:

#config file
[mqtt]
RMSCFG=/home/pi/source/RMS/RMS_config
BROKER=192.168.1.158
USERNAME=admin
PASSWORD=XXXX
MQPORT=8883
CAFILE=

I have checked the path and content of the RMS_config file. It looks all fine. However if I run the bash script I get:

(vRMS) pi@raspberrypi:~ $ /home/pi/source/rms_mqtt/logMeteorStats.sh
/home/pi/source/rms_mqtt/config.ini
Traceback (most recent call last):
  File "/home/pi/source/rms_mqtt/sendToMQTT.py", line 164, in <module>
    sendToMqtt()
  File "/home/pi/source/rms_mqtt/sendToMQTT.py", line 88, in sendToMqtt
    cfg = cr.parse(os.path.expanduser(rmscfg))
  File "/home/pi/vRMS/lib/python3.7/site-packages/RMS-0.1-py3.7-linux-armv7l.egg/RMS/ConfigReader.py", line 680, in parse
    raise RuntimeError('Unknown config file name: {}'.format(os.path.basename(path)))
RuntimeError: Unknown config file name: RMS_config

[markmac99]

The RMS config file is named .config and its a hidden file in the /home/pi/source/RMS folder The name RMS_Config is just a desktop-shortcut name to make it easier for camera operators to find it.

[WirthmU]

Thank you for your patience. I set name and path correctly now. To my surprise the RMS .config file has been changed and set to the default values. I have not changed this since 2022, but on the globalmeteornetwork.org site where my cameras are published the station code still was the correct one. So now everything in the .configfile is right again and at least the logMeteorStats.sh publishes mqtt with the right name.

Unfortunately the broker still misses the logPiStats:

[markmac99]

As i mentioned earlier, RMS has a self-update process that runs every time the Pi is rebooted. This process temporarily replaces your .config file with the default one, then restores if afterwards. If the update process is interrupted, for example the RMS window is closed or the Pi is rebooted unexpectedly, then the file may not get restored properly. Additionally, if you examine the config file while the update is in progress, you will see the default one. Its therefore important to wait till the RMS update process completes before checking the config or making any changes. The update normally takes about five minutes and should always be allowed to complete.

[markmac99]

On the issue with logPiStats, i forgot to add TLS support to the sendOtherData() function :) Done now

[markmac99]

ps if you find your config file keeps getting changed you should ask for advice in the GlobalMeteorNetwork group in groups.io. https://globalmeteornetwork.groups.io/g/main/topics

[WirthmU]

Sounds logical, all this. I will keep an eye now on the RMS .config file.

I just tried to update using the update procedure. It fails with:

(vRMS) pi@raspberrypi:~/source/rms_mqtt $ git stash
config.ini: needs merge
config.ini: needs merge
config.ini: unmerged (996da84a55d8c16d1f6cd48cb0dd0475e65b214e)
config.ini: unmerged (7aa800dbc233dbb93f307faf703145246725a9b2)
config.ini: unmerged (50703b8b9609f2c800b8314d95392a058cfa0ee6)
fatal: git-write-tree: error building trees
Cannot save the current index state

[markmac99]

Apologies but this isn't a problem with my code, so i will have to leave you to google it.

[WirthmU]

I allow myself to re-open with a request and and idea. But first - I fixed the git problem using the command git reset --mixed.

Request: since I do have several RPI's running RMS and I would like to use your MQTT-program for those, the MQTT base-topic "raspberrypi" will be redundant. Would it be possible to use the camera code instead (i.e. ch0002)?

Idea: it would be nice to have a continuous star count during the night (ever y five minutes). It would be best if the readout would be part of the logPiStats.sh script. This would be then like a cheap Sky Quality Meter (SQM). I am already using this with a camera that is running indi-allsky (see picture below).

[markmac99]

I've written the code so that the hostname of each Pi is used as the base topic. So, if you change the hostname of each station to match its RMS ID, then you will achieve what you want. It will also make it easier to identify which station you're on.

The purpose of this module was to demonstrate how to post some basic data to MQ. I'm therefore not going to make any further changes. However, in another of my repos, i have functionality to post the star count to MQ every five minutes. Take a look at trackStarCount.sh in that repository, and then take a look at the sendStarCountToMqtt() function in sendToMQTT.py in that repo, though you could also easily adapt sendOtherData() in this repo to include the starcount. Finally, I'm closing this because I've fixed the origial issue, and you're now starting to make feature requests on the bug report issue. If you have any new requests please open a new ticket, but as noted i will only be fixing bugs.