allsky_publishdata MQTT issue

starhopper-de commented 5 months ago

Describe the bug I tried to use allsky_publishdata with MQTT. Configure in Module Manger: Server: 192.168.8.1 MQTT Port: 1883 Topic: Allsky Username and Passwort

I run tcpdump to see what is happen. It holds a tcp connection with my server, but sends a different MQTT Topic and states a data I guess from a different module. The clearsky module is definitely not running. What should I do?

Expected behaviour A clear and concise description of what you expected to happen.

Screenshots tail -f /var/log/allsky.log Feb 16 23:57:14 allsky allsky[819]: STARTING EXPOSURE at: 2024-02-16 23:57:14 @ 30.0 sec Feb 16 23:57:49 allsky allsky[819]: > Saving NIGHT image 'image-20240216235714.jpg' Feb 16 23:58:01 allsky allsky[2326]: ERROR: Module allsky_publishdata.py failed on line 224 - [Errno 104] Connection reset by peer....

TCPdump:allsky.lan.50835 > 192.168.8.1.1883: Flags [S], cksum 0x2ff4 (correct), seq 3238881066, win 64240, options [mss 1460,sackOK,TS val 2326663658 ecr 0,nop,wscale 6], length 0 192.168.8.1.1883 > allsky.lan.50835: Flags [S.], cksum 0xc2d1 (correct), seq 969868372, ack 3238881067, win 28960, options [mss 1286,sackOK,TS val 1200452061 ecr 2326663658,nop,wscale 8], length 0 allsky.lan.50835 > 192.168.8.1.1883: Flags [.], cksum 0x5dfb (correct), seq 1, ack 1, win 1004, options [nop,nop,TS val 2326663700 ecr 1200452061], length 0 allsky.lan.50835 > 192.168.8.1.1883: Flags [P.], cksum 0x4c4a (correct), seq 1:224, ack 1, win 1004, options [nop,nop,TS val 2326663700 ecr 1200452061], length 223 192.168.8.1.1883 > allsky.lan.50835: Flags [.], cksum 0x606c (correct), seq 1, ack 224, win 118, options [nop,nop,TS val 1200452099 ecr 2326663700], length 0 192.168.8.1.1883 > allsky.lan.50835: Flags [R.], cksum 0x6068 (correct), seq 1, ack 224, win 118, options [nop,nop,TS val 1200452099 ecr 2326663700], length 0 {"available":{"clearsky":{"module":"allsky_clearsky.py","metadata":{"name":"Clear Sky Alarm","description":"Clear Sky Alarm","module":"allsky_clearsky","version":"v1.0.0","events":{"0":"day","1":"night"},"experimental":true,"arguments":{"detectionThreshold":0.55,"distanceThreshold":20,"annotate":"false","template1":6,"mask":"","debug":"false","clearvalue":10,"roi":"","roifallback":5,"mqttenable":"False","mqttbroker":"","mqttport":1883,"mqttusername":"","mqttpassword":"","mqtttopic":"SKYSTATE","debugimage":""},"argumentdetails":{"roi":{"required":"true","description":"Region of Interest","help":"The area of the image to check for clear skies. Format is x1,y1,x2,y2","type":{"fieldtype":"roi"}},"roifallback":{"required":"true","description":"Fallback %","help":"If no ROI is set then this % of the image, from the center will be used","type":{"fieldtype":"spinner","min":1,"max":100,"step":1}},"clearvalue":{"required":"true","description":"Clear Sky","help":"If more than this number of stars are found the sky will be considered clear","type":{"fieldtyp [|http] allsky.lan.43977 > 192.168.8.1.1883: Flags [S], cksum 0xa1bd (correct), seq 2178699516, win 64240, options [mss 1460,sackOK,TS val 2326728777 ecr 0,nop,wscale 6], length 0 192.168.8.1.1883 > allsky.lan.43977: Flags [S.], cksum 0x7ec7 (correct), seq 209643806, ack 2178699517, win 28960, options [mss 1286,sackOK,TS val 1200517174 ecr 2326728777,nop,wscale 8], length 0 allsky.lan.43977 > 192.168.8.1.1883: Flags [.], cksum 0x1a01 (correct), seq 1, ack 1, win 1004, options [nop,nop,TS val 2326728803 ecr 1200517174], length 0 allsky.lan.43977 > 192.168.8.1.1883: Flags [P.], cksum 0x46c1 (correct), seq 1:224, ack 1, win 1004, options [nop,nop,TS val 2326728803 ecr 1200517174], length 223

Allsky Information:

OS: [name, 64/32bit and Full/lite]
Allsky Version: Version: v2023.05.01_04, allsky-module repository cloned at 16.02.24
Pi model: [Include RAM] Pi 3 Model A+ Revision 1.0 (512 MB)
Camera: [Full details of the camera being used] Raspberry PI HQ

Sensors If using sensors please outline the sensors being used, include i2c addresses if applicable

Additional context Add any other context about the problem here.

Alex-developer commented 5 months ago

Are you running the local broker thru ssl?

Currently the module requires a secure connection to the broker. I can easily add a non secure connection if needed

starhopper-de commented 5 months ago

No, our observatory has an openvpn. It would be great to add a non securce connection without websocket 9001

Alex-developer commented 5 months ago

No, our observatory has an openvpn. It would be great to add a non securce connection without websocket 9001

No problem I will add an option to disable all of the ssl connectivity.

Alex-developer commented 4 months ago

I have added support for non SSL connections. Few notes

I have tested this with mosquitto running locally and HiveMQ onlne.
You can just copy the allsky_publishdata.py from this repo into /opt/allsky/modules - Or you can reinstall it
You will have to remove the module from the flow, save the flow, refresh the flow manager then readd the module

New options

'Use Secure Connection' Determines if SSL (TLS) is used. If it is the port number will typically be 8883 if NOT selected the port number will typically be 1883
'Loop Delay(s)' The amount of time to wait for MQTT messages to be published. Unless you have any issues just leave this at the default.

Please let me know how you get on

Cheers

Alex

starhopper-de commented 4 months ago

Thanks, I am happy, it works fine. Very fast work. Please let me know why data publishing is only usable in day or night. I would rather see export features in periodic jobs?

Alex-developer commented 4 months ago

Thanks, I am happy, it works fine. Very fast work. Please let me know why data publishing is only usable in day or night. I would rather see export features in periodic jobs?

Added the periodic flow now so please give it a try

starhopper-de commented 4 months ago

I have given it a try. Now data comes periodically. Not all, see my output. Data from my modules allkskydew and pistatus are exported. But I am not able to export eg. MoonIllumination. At the moment, all data I am interested in, is exported. You can close this issue. Thanks Thomas

Allskyperiodic Log:

Feb 20 22:39:40 allsky allskperiodic[20904]: INFO: PI Status Data Written
Feb 20 22:39:41 allsky allskperiodic[20904]: INFO: Turning Heater on using pin 4
Feb 20 22:39:41 allsky allskperiodic[20904]: INFO: Temperature within limit temperature 6.78, limit 5, dewPoint 3.25
Feb 20 22:39:41 allsky allskperiodic[20904]: INFO: Sensor SHT31 read. Temperature 6.78 Humidity 78.23 Relative Humidity None Dew Point 3.25 Heat Index -17.32 Pressure None Altitude None
Feb 20 22:39:41 allsky allskperiodic[20904]: ERROR: Cannot locate environment variable DAY_OR_NIGHT specified in the extradata
Feb 20 22:39:41 allsky allskperiodic[20904]: ERROR: Cannot locate environment variable CURRENT_IMAGE specified in the extradata
Feb 20 22:39:41 allsky allskperiodic[20904]: INFO: MQTT - Published to MQTT on channel: /SternwarteXXXX/Allsky

{"allskydew": {"AS_DEWCONTROLSENSOR": "SHT31", "AS_DEWCONTROLAMBIENT": "6.78", "AS_DEWCONTROLDEW": "3.25", "AS_DEWCONTROLHUMIDITY": "78.23", "AS_DEWCONTROLHEATER": "On"}, "pistatus": {"AS_DISKSIZE": "116.95G", "AS_DISKUSAGE": "43.21G", "AS_DISKFREE": "68.95G", "AS_CPUTEMP": "31.1", "AS_THROTTLEDBINARY": "0x0", "AS_TSTAT0": "False", "AS_TSTAT1": "False", "AS_TSTAT2": "False", "AS_TSTAT3": "False", "AS_TSTAT16": "False", "AS_TSTAT17": "False", "AS_TSTAT18": "False", "AS_TSTAT19": "False", "AS_TSTATSUMARYTEXT": "", "AS_CLOCKARM": "1.4", "AS_CLOCKCORE": "0.4", "AS_CLOCKISP": "0.0", "AS_CLOCKV3D": "0.3", "AS_CLOCKUART": "0.048", "AS_CLOCKPWM": "0.0", "AS_CLOCKEMMC": "0.199999", "AS_CLOCKPIXEL": "0.000338", "AS_CLOCKVEC": "0.108", "AS_CLOCKHDMI": "0.163683", "AS_CLOCKDPI": "0.0", "AS_VOLTAGECORE": "1.375", "AS_VOLTAGESDRAM_C": "1.25", "AS_VOLTAGESDRAM_I": "1.25", "AS_VOLTAGESDRAM_P": "1.225"}, "CAMERA_TYPE": "RPi", "FULL_FILENAME": "image.jpg", "ALLSKY_VERSION": "v2023.05.01", "utc": 1708465180}

starhopper-de commented 4 days ago

Sorry, but now I am keen on getting DAY_OR_NIGHT via MQTT. I got the newest allsky_publishdata.py but it still does not work. Is it possible for you to get this working. Thomas

Jul 13 12:35:19 allsky allskperiodic[21379]: INFO: PI Status Data Written Jul 13 12:35:19 allsky allskperiodic[21379]: INFO: Turning Fans OFF Jul 13 12:35:19 allsky allskperiodic[21379]: INFO: CPU Temp is 50.1 and lower then set limit of 60, Fans are Off via fan pin 14 Jul 13 12:35:19 allsky allskperiodic[21379]: INFO: Rain State: Its Not Raining Jul 13 12:35:19 allsky allskperiodic[21379]: INFO: Temperature outside limit temperature 23.32, limit 5, dewPoint 11.58 Jul 13 12:35:19 allsky allskperiodic[21379]: INFO: Turning Heater off using pin 4 Jul 13 12:35:19 allsky allskperiodic[21379]: INFO: Sensor SHT31 read. Temperature 23.32 Humidity 47.63 Relative Humidity None Dew Point 11.58 Heat Index -8.01 Pressure None Altitude None Jul 13 12:35:19 allsky allskperiodic[21379]: ERROR: Cannot locate environment variable DAY_OR_NIGHT specified in the extradata Jul 13 12:35:19 allsky allskperiodic[21379]: ERROR: Cannot locate environment variable MOON_ILLUMINATION specified in the extradata Jul 13 12:35:19 allsky allskperiodic[21379]: ERROR: Cannot locate environment variable AS_MOON_ILLUMINATION specified in the extradata Jul 13 12:35:19 allsky allskperiodic[21379]: INFO: MQTT - Published to MQTT on channel: Sternwarte/Allsky

AllskyTeam / allsky-modules

allsky_publishdata MQTT issue #92