VRM portal isn't updated when shm-et340 is running

[howetech]

This tool is working perfectly on the local device and network and correctly reports all the data from the SMA meter for use in an ESS system.

The problem is, whenever this program is running, the Victron online VRM portal stops getting updates from the local device. As soon as I terminate shm-et340, VRM portal instantly starts getting updates and working as normal.

Is there a problem with some of the data generated by this tool that prevents the VRM portal from accepting any device stats / updates?

[sevelen]

I´ve the same issue. So far no solution.

[mitchese]

Very odd, in the last week I think this just started to affect me. Until last week, the VRM Portal worked as expected for me, with everything showing correctly. Suddenly and only occasionally the large numbers would be incorrect. I think it may be related to the (missing, deprecated) value for total Amps on the Gridmeter. I left this out because according to https://github.com/victronenergy/venus/wiki/dbus#grid-and-genset-meter, the total Amps (and Voltage) is Deprecated, but it (along with the nonsensical Voltage) are the only things not implemented.

Can you confirm the behaviour is the same as what I am seeing, namely the values in Overview are correct

but when you view the details page

the graph data is correct but the values are possibly 0? I can't recreate it right now for screenshots, because it is now working (with no changes), but I have seen where my battery is stuck at a certain percentage, and the values for Load/Grid/Inverter are all wrong (either stuck on a value from last week, or stuck at 0/nonexistent)

[sevelen]

The main issue for me is, if i use the latest version the VRM portal will not going into realtime mode anymore. if i change the version back its working directly

[mitchese]

ok that sounds different to the issues I've seen over the last few days. I grabbed the latest from github to make sure I'm not using a special build root@victronvenus:~# md5sum shm-et340-gitub 5e24d2356689e30f38622b1fca82ce56 shm-et340-gitub

and it does work in my setup including realtime updates - I assume you mean the little realtime in the portal like this, right?

[mitchese]

Does it log anything - more specifically do you see anything like a lot of 'The protocol ID didn't match 0x6069, it's not a meter update. ProtocolID: "? I wonder if the protocol ID of your SHM is different and it's rejecting all the updates in the newest version

The realtime should be provided by data over the MQTT connection - have you tried looking at what mosquitto is doing? This should provide an insight into what it's doing: strace -p $(pgrep mosquitto) -e read,write -s 512 (there may be an easier debug option which can be enabled in mosquitto, but strace gives a good indication of if anything is happening)

[sevelen]

Yes you are right thats what i mean, by the old version i can track the messages comming fast and the VRM switched to realtime.

If i do the same by the latest release version near than nothing refresh by the MQTT and VRM does´t to into realtime.

strace: Process 1291 attached read(13, "\27\3\3\0003", 5) = 5 read(13, "\212\362c\232\367Kxw\246\360^\265\240\3448\244\5\354\202\35Qe\244\245\265\312\232\23\308\355\241\235\357\360C\202\35\271\206c\331\371Y\6\352GD\240\212W", 51) = 51 read(13, "\27\3\3\0003", 5) = 5 read(13, "\351\352\"(R,\324$\332|\302\254$E\355.\317\314\3430\373|=+\217'GCXK\200\256cOR\2\21\370\270\233!\v\24\221IM\272\20\367\2541", 51) = 51 write(12, "\27\3\3\0\23\3521u\34u8\222\316\312S\20\202z\377\274\273cz\230", 24) = 24 read(12, "\27\3\3\0\23", 5) = 5 read(12, "/\360\350N\217B\3]c\221\316\355\321\233\224)P\205A", 19) = 19 write(13, "\27\3\3\0\23\27\270{\17\220<c\272?\207z\16@\233}\236\3456S", 24) = 24 read(14, "\300", 1) = 1 read(14, "\0", 1) = 1 write(14, "\320\0", 2) = 2 read(13, "\27\3\3\0003", 5) = 5 read(13, "\357\f\274\335\352\3\256:\313\232\334\216\330\372\1\311\32\306\365\266\273\234\253\214-\256\234\300\232z\33\367k\321[\25\356\35\\U\6\177\366:\17L\3072\202\2069", 51) = 51 read(13, "\27\3\3\0\23", 5) = 5 read(13, "\267\256\317\263);\277\260\2413M\324\351\7\227=\311\355 ", 19) = 19 read(13, "\27\3\3\0003", 5) = 5 read(13, "#ZQs \301\277@\276o/\300\f\376s>\302\206\362\177\36\22\241\367\5\321\312\310tmI\270\206\243\211\177\205\10\336\210\231\226}6\341\31\333u\364l\\", 51) = 51

[mitchese]

Ok and what about the logs from it while it is running if you start it manually? The only difference between the versions is, it tries to reject implausible meter updates to filter out some weirdness which came about in a recent update to SHM. Every time it drops a "meter update" which fails to be identified as a meter update, it will log why it dropped it ... what I'm thinking is happening is for some reason your SHM is different from mine (most likely that protocol id not being 6069).

Also maybe it's assuming a single phase update is the same byte size as a threephase - I have 3phase, so also the "packet too small" may be too aggressively droppings stuff. This should log too with something implausibly small in the message

When you start it, it drops all messages because (reason), but because you have a valid meter "attached" it stops sending updates to the portal. This is theory though, and lives and dies by what is logged.

[Gardun954]

I'm very new to Venus OS, programming and stuff on Raspberry, but with the help of all your scripts I had the HM2 faker running on the Raspberry in half an hour. Yesterday I had it running for multiple hours manually started in the shell. This worked with the Victron VRM perfectly and it updated in realtime as soon as I started the script manually in the shell (I don't have any other Victron devices yet, so the only inputs are the values from the HM2 faker at the moment).

Today I added the rc.local file to implement the autostart of the script. The moment when I rebooted the Raspberry, the VRM update stopped. After I deleted the rc.local file and rebooted again, the VRM updated in realtime again (after starting the script manually). So it for sure has something to do with the autostart (maybe overriding the autostart of the VRM update?)

Maybe this helps any of you guys to fix the issue, as I don't know any of the linux commands (shame on me), this is the only help that I can offer :).

[Gardun954]

Hey guys, I think I just found a solution although I completely don't know what I am doing.

When researching for "linux execute commands at startup", I realized that in all the other scripts, the scripts started with #!/bin/bash in the first line instead of !#/bin/bash , which can be found in the Theard by Schnema "MultiPlus 2 GX run own script at startup" and which screwed up the VRM portal realtime update (when the following script was run in /data/rc.local): !#/bin/bash sleep15 setsid /data/home/root/shm-et340 > /dev/null 2>/dev/null &

As I found this strange and I thought that a script-line which you can't find in the internet, is probably a buggy script-line, i just deleted it. Now the run at startup works as well as the realtime update.

So for me, this works for automatic run at startup and not screwing up the realtime update for the VRM portal: create file in /data/rc.local with the following two lines only: sleep 15 setsid /data/home/root/shm-et340 > /dev/null 2>/dev/null &

Please correct me if I have done something completely stupid (I really don't know what I am doing), but as it works, I'll keep it like that.

[Knedox]

As all of you i had the same problem that real time updates in vrm portal are not working anymore. I had a deeper look today (but no good solution yet)

The problem is that some dbus value provided by the shm-et340 seems fishy

root@raspberrypi2:~# /opt/victronenergy/dbus-mqtt/dbus_mqtt.py --init-broker -------- dbus_mqtt, v1.30 is starting up -------- INFO:logger:Loglevel set to INFO INFO:mosquitto_bridge_registrator:[InitBroker] Reading config file INFO:mosquitto_bridge_registrator:[InitBroker] Registering CCGX at VRM portal INFO:mosquitto_bridge_registrator:[InitBroker] Not updating config file and not restarting Mosquitto, because config is correct. INFO:mosquitto_bridge_registrator:[InitBroker] Registration successful INFO:dbus_mqtt:[Scanning] service: com.victronenergy.qwacs /opt/victronenergy/dbus-mqtt/dbus_mqtt.py:402: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead logging.warn('[Scanning] {} does not provide an item listing'.format(service)) WARNING:dbus_mqtt:[Scanning] com.victronenergy.qwacs does not provide an item listing INFO:dbus_mqtt:[Scanning] service: com.victronenergy.hub4 INFO:dbus_mqtt:[Scanning] service: com.victronenergy.settings INFO:dbus_mqtt:[Scanning] service: com.victronenergy.system INFO:dbus_mqtt:[Scanning] service: com.victronenergy.fronius INFO:dbus_mqtt:[Scanning] service: com.victronenergy.pvinverter.pv_xxx INFO:dbus_mqtt:[Scanning] service: com.victronenergy.grid.cgwacs_ttyUSB0_di30_mb1 Traceback (most recent call last): File "/opt/victronenergy/dbus-mqtt/dbus_mqtt.py", line 397, in _scan_dbus_service items = self._get_dbus_value(service, '/') File "/opt/victronenergy/dbus-mqtt/dbus_mqtt.py", line 533, in _get_dbus_value return self._dbus_conn.call_blocking(service, path, None, 'GetValue', '', []) File "/usr/lib/python3.8/site-packages/dbus/connection.py", line 652, in call_blocking reply_message = self.send_message_with_reply_and_block( dbus.exceptions.DBusException: org.freedesktop.DBus.Error.UnknownMethod: Unknown / invalid method

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/victronenergy/dbus-mqtt/dbus_mqtt.py", line 602, in main() File "/opt/victronenergy/dbus-mqtt/dbus_mqtt.py", line 584, in main handler = DbusMqtt( File "/opt/victronenergy/dbus-mqtt/dbus_mqtt.py", line 217, in init self._scan_dbus_service(service) File "/opt/victronenergy/dbus-mqtt/dbus_mqtt.py", line 401, in _scan_dbus_service self._introspect(service, device_instance, '/', publish) File "/opt/victronenergy/dbus-mqtt/dbus_mqtt.py", line 441, in _introspect self._introspect(service, device_instance, p, publish=publish) File "/opt/victronenergy/dbus-mqtt/dbus_mqtt.py", line 441, in _introspect self._introspect(service, device_instance, p, publish=publish) File "/opt/victronenergy/dbus-mqtt/dbus_mqtt.py", line 423, in _introspect value = self._dbus_conn.call_blocking(service, path, None, 'Introspect', '', []) File "/usr/lib/python3.8/site-packages/dbus/connection.py", line 652, in call_blocking reply_message = self.send_message_with_reply_and_block( dbus.exceptions.DBusException: org.freedesktop.DBus.Error.UnknownMethod: Unknown / invalid method

The workaround as written everywhere is to start the shm-et340 after the mqtt broker in rc.local:

(sleep 25 && setsid /data/home/root/shm-et340 > /dev/null 2>/dev/null) &

[mitchese]

I think this is more of a load issue, which I don't have a good solution for. I set my update interval to 1 minute (instead of the default 15m), and realtime was sometimes working and sometimes not (falling back to 1 minute updates in the vrm portal).

Looking at the venus GX, the poor thing has a load average of 4.00 (single core, slow CPU). shm-et340 isn't the culprit (and is responsible for around 8% of the usage) - but all other things running on it I believe mean it's simply not powerful enough.

I tested my theory by stopping the gui svc -d /service/gui

and a few other things in /service that were not relevant to my installation. This freed up enough processing time and magically the realtime updates started happening.

If you accidentally stopped something you needed you can restart with svc -u /service/gui

and see what services are there with ls /service and top to see who's using much time.

With this confirmed, I switched over to using a raspberry pi 4 - this went from a load average of 4.0 on the venus gx with a single core to 0.8 on a raspberry pi with 4 cores - so huge improvement.

The binaries work exactly the same on a rpi, no recompile or changes needed.