search

freakent / dbus-mqtt-devices

A driver for Victron VenusOS GX devices to support use of dbus-mqtt/flashmq-mqtt
MIT License
111 stars 24 forks source link

Error during install script #12

Closed Hofyyy closed 2 years ago

Hofyyy commented 2 years ago

Hello,

I got the following error:

`root@ccgx:~# cd root@ccgx:~# cd /data/ root@ccgx:/data# mkdir -p drivers root@ccgx:/data# cd drivers/ root@ccgx:/data/drivers# wget -O dbus-mqtt-devices.zip https://github.com/freakent/dbus-mqtt-devices/archive/refs/tags/v0.4.1.zip --2022-04-30 07:20:55-- https://github.com/freakent/dbus-mqtt-devices/archive/refs/tags/v0.4.1.zip Resolving github.com... 140.82.121.3 Connecting to github.com|140.82.121.3|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/freakent/dbus-mqtt-devices/zip/refs/tags/v0.4.1 [following] --2022-04-30 07:20:55-- https://codeload.github.com/freakent/dbus-mqtt-devices/zip/refs/tags/v0.4.1 Resolving codeload.github.com... 140.82.121.9 Connecting to codeload.github.com|140.82.121.9|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/zip] Saving to: ‘dbus-mqtt-devices.zip’

dbus-mqtt-devices. [ <=> ] 21.47K --.-KB/s in 0.01s

2022-04-30 07:20:56 (2.15 MB/s) - ‘dbus-mqtt-devices.zip’ saved [21983]

root@ccgx:/data/drivers# unzip dbus-mqtt-devices.zip Archive: dbus-mqtt-devices.zip creating: dbus-mqtt-devices-0.4.1/ inflating: dbus-mqtt-devices-0.4.1/.gitignore inflating: dbus-mqtt-devices-0.4.1/.gitmodules inflating: dbus-mqtt-devices-0.4.1/LICENSE inflating: dbus-mqtt-devices-0.4.1/README.md creating: dbus-mqtt-devices-0.4.1/bin/ creating: dbus-mqtt-devices-0.4.1/bin/service/ creating: dbus-mqtt-devices-0.4.1/bin/service/log/ inflating: dbus-mqtt-devices-0.4.1/bin/service/log/run inflating: dbus-mqtt-devices-0.4.1/bin/service/run.tmpl inflating: dbus-mqtt-devices-0.4.1/bin/setup.sh inflating: dbus-mqtt-devices-0.4.1/bin/uninstall.sh inflating: dbus-mqtt-devices-0.4.1/dbus_mqtt_devices.py inflating: dbus-mqtt-devices-0.4.1/device.py inflating: dbus-mqtt-devices-0.4.1/device_manager.py inflating: dbus-mqtt-devices-0.4.1/device_service.py inflating: dbus-mqtt-devices-0.4.1/device_service_config.py creating: dbus-mqtt-devices-0.4.1/ext/ creating: dbus-mqtt-devices-0.4.1/ext/dbus-mqtt/ creating: dbus-mqtt-devices-0.4.1/ext/velib_python/ inflating: dbus-mqtt-devices-0.4.1/requirements.txt inflating: dbus-mqtt-devices-0.4.1/services.yml creating: dbus-mqtt-devices-0.4.1/test-data/ inflating: dbus-mqtt-devices-0.4.1/test-data/device-status.json inflating: dbus-mqtt-devices-0.4.1/test-data/pvinverter-simulator.py inflating: dbus-mqtt-devices-0.4.1/test-data/tank-level.json inflating: dbus-mqtt-devices-0.4.1/test-data/tank-simulator.py inflating: dbus-mqtt-devices-0.4.1/test-data/temperature.json creating: dbus-mqtt-devices-0.4.1/tests/ inflating: dbus-mqtt-devices-0.4.1/tests/test_validation.py root@ccgx:/data/drivers# ./dbus-mqtt-devices-0.4.1/bin/setup.sh Setup dbus-mqtt-devices in /data/drivers/dbus-mqtt-devices-0.4.1 started Ensure Python's Pip is installed pip 20.0.2 from /usr/lib/python3.8/site-packages/pip (python 3.8) Pip install module dependencies Collecting PyYAML==6.0 Downloading PyYAML-6.0.tar.gz (124 kB) |████████████████████████████████| 124 kB 978 kB/s Installing build dependencies ... error ERROR: Command errored out with exit status -9: command: /usr/bin/python /usr/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /var/volatile/tmp/pip-build-env-jqwzyv56/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel Cython cwd: None Complete output (7 lines): Collecting setuptools Downloading setuptools-62.1.0-py3-none-any.whl (1.1 MB) Collecting wheel Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB) Collecting Cython Downloading Cython-0.29.28-py2.py3-none-any.whl (983 kB) Installing collected packages: setuptools, wheel, Cython

ERROR: Command errored out with exit status -9: /usr/bin/python /usr/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /var/volatile/tmp/pip-build-env-jqwzyv56/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel Cython Check the logs for full command output. Set up Victron module libraries Set up device service to autorun on restart grep: /data/rc.local: No such file or directory Setup dbus-mqtt-devices complete`

Hofyyy commented 2 years ago

Device: Color Control GX Firmware: v.2.85 Clean Image

freakent commented 2 years ago

Both @stundenblume and myself are running this on Raspberry Pi's. Last night I tried to install on a CCGX device. The installation caused the device to reset and will no longer boot. I recommend you do not try to install this on CCGX until I find out why.

Hofyyy commented 2 years ago

Thank you for the information!

freakent commented 2 years ago

I managed to get the driver to install on CCGX but the MQTT broker is incredibly unstable. The connection from the driver to MQTT is continuously disconnecting and reconnecting. It maybe that CCGX just isn't up to the job. Which is a shame because my whole reason for writing this was to deploy on CCGX. I'll keep investigating.

Hofyyy commented 2 years ago

Thats sounds great and bad :-)

gharshaw commented 2 years ago

Bummer. I was just about to try this on my CCGX v.2.85. I'm reading bluetooth temp sensors and bluetooth SeeLevel Tank sensors via Node-Red PI into the CCGX with MQTT. I've been using multiple "dummy d-bus services" to add these devices. Hoping you are able to find a solution as I like the idea of the self registration. Thanks for your work on this.

Hofyyy commented 2 years ago

My tasmota Device is Running perfectly. So wie need a solution here 😜

freakent commented 2 years ago

Hey, I've been experimenting! 😀 If anyone else has any ideas I'm listening.

freakent commented 2 years ago

In release 0.41 I am using pip to install PYyaml. On my CCGX as soon as I tried to install PYyaml using pip, the device reset and would no longer boot (I was able to recover the device using a firmware on an SD card - twice!). I guess it was something to do with compiling PYyaml to native code. I looked around for a pure python alternative only to end up back at PYyaml. It turns out you can install a pure python version of PYyaml. I did this on my CCGX and it installed fine. I then ran dbus-Mqtt-devices directly from the command line and found it was continuously connecting and disconnecting with a protocol error (rc = 1). I have been comparing how dbus-mqtt (from Victron) and dbus-mqtt-devices are run. So far I haven't found the magic switch that stabilises things.

gharshaw commented 2 years ago

I found this comment in an MQTT troubleshooting blog about rc=1 -- Maybe it will shed some light :-) "this sometimes happens, when one program is running twice an connects from the same IP with the same client ID. The broker than closes the "old connection" which is still held by the process running in the background."

gharshaw commented 2 years ago

I see that dbus-mqtt (from Victron) has a MAX_TOPIC_AGE = 60s keep-alive arg in dbus-mqtt. I don't see the arg in your script.

parser.add_argument('-k', '--keep-alive', default=MAX_TOPIC_AGE, help='keep alive interval in seconds', type=int)

I don't know if this is the issue - just throwing out thoughts that might shed more light.

freakent commented 2 years ago

Thanks, very useful. Two things I can investigate. If you find any more let me know.

stundenblume commented 2 years ago

Hey, I've been experimenting! grinning If anyone else has any ideas I'm listening.

Hi Martin, sorry I don't own a CCGX nor I'm a experienced programmer. Sorry! Sebastian

gharshaw commented 2 years ago

Martin, BTW, I'm willing to help you with any testing on my CCGX

Gene

freakent commented 2 years ago

Hi Gene,

I think the thing to do here is to get the installation working then we can see if the stability issue is just on my device. I do you want to remind you that my CCGX completely crashed and I had to re-flash a standard firmware by hand to get it working again. I think we can avoid that happening again but I strongly suggest that you have a recovery SD card standing by and that you make a note of any settings you don't want to loose.

Here are Victron's instructions for manual upgrade, just in case. https://www.victronenergy.com/live/ccgx:firmware_upgrade_to_v2 These instructions will revert your ccgx to an older version, but you then perform an online upgrade to bring it back up to the latest version.

Are you happy with this?

freakent commented 2 years ago

setup-ccgx.zip

Given my large disclaimer above, I have attached a new setup-ccgx.sh script (I had to zip it up as Github does not allow .sh files). Are you comfortable unzipping the attached file and copying the .sh script into your /data/drivers/dbus-mqtt-device-0.4.0.3/bin directory ?

You will probably need to ensure you have the right permissions set on the script. chmod +x dbus-mqtt-device-0.4.0.3/bin/setup-ccgx.sh

Then you can run this modified set up script, ./dbus-mqtt-devices-0.4.1/bin/setup-ccgx.sh

Hofyyy commented 2 years ago

@freakent I can do testing too. Because my "Color Control GX" is not used until now and I have no problem with flashing stock if needed. My real installation will start next weekend.

I did now:

System is online again with remote console and SSH

What can I test? Or do you need VPN access?

I will now change my tasmota to the real MQTT Broker :-) and play a little bit.

System is still alive. Service is running

gharshaw commented 2 years ago

Hi Gene,

I think the thing to do here is to get the installation working then we can see if the stability issue is just on my device. I do you want to remind you that my CCGX completely crashed and I had to re-flash a standard firmware by hand to get it working again. I think we can avoid that happening again but I strongly suggest that you have a recovery SD card standing by and that you make a note of any settings you don't want to loose.

Here are Victron's instructions for manual upgrade, just in case. https://www.victronenergy.com/live/ccgx:firmware_upgrade_to_v2 These instructions will revert your ccgx to an older version, but you then perform an online upgrade to bring it back up to the latest version.

Are you happy with this?

Yes - I recently had to restore back to 2.18 with Sdcard due to ethernet interface remaining "unplugged"

freakent commented 2 years ago

Take a look at the Troubleshooting instructions.

https://github.com/freakent/dbus-mqtt-devices#troubleshooting

I would like to see the contents of your log file /var/log/dbus-mqtt-devices/current

If it is constantly disconnecting and re connecting it should be very evident from the log.

freakent commented 2 years ago

@Hofyyy I think I see an extra ext directory inside your main ext directory. That would suggest you ran the set up script from the wrong folder. Try following the steps in the install instructions in the readme, except using the setup-ccgx.sh.

Hofyyy commented 2 years ago

@freakent All is running fine now.

gharshaw commented 2 years ago

@freakent Marty, The driver is so far running very stable. I have not seen it restart at all in the logs. I have 5 temp sensors and 4 tank sensors being registered and updated. I do see one issue after I reboot the CCGX. I need to re-register my <device//Status> payload and when I do, the local settings like "custom name" get overwritten to the default of "My Tank Sensor" or "My Temperature Sensor. Is this expected? Do you perhaps update these custom names dynamically by publishing a mqtt message to the custom name value?

Awesome work!!

Gene

freakent commented 2 years ago

Well it's good to know it's running stable.

You need to register with the status message every time you connect to Mqtt. Especially if you reboot the GX device. That is a requirement of Victron's DBUS implementation.

When you reregister with the same details you should get the same device Id. If you get the same device id then it should retrieve the the previously saved custom name. Check the log file and upload a copy to an issue if this continues to be a problem.

You should be setting the custom name from the menu in the GX console.

gharshaw commented 2 years ago

Here is log file after rebooting and running the re-register. I am getting the same device ID's unfortunately local settings are getting updated to what appears to be the values in the services.yml file. log.txt

gharshaw commented 2 years ago

@freakent

@gharshaw Gene, I just noticed you are running 0.4.0.1, can you update to the latest 0.4.0.3 ?

I think I'm running dbus_mqtt_devices, v0.4.1 - the latest release. starting dbus-mqtt-devices * @400000006272e42b0fe3d154 -------- dbus_mqtt_devices, v0.4.1 is starting up --------**

freakent commented 2 years ago

Gene, Sorry, please ignore that suggestion.

freakent commented 2 years ago

It sounds like the ccgx installation solution is working. So I will close this issue for now. I am preparing a new 0.5 release with the setup-ccgx.sh script.