Serial Port - Modbus Packet Loss with RPi Zero W

[erfahnoe]

Use the template below if you have an issue or want to report a bug. If you have a question or a feature request you can ignore the questions below.

NOTE: If you are having issues with your serial connection, please read this page before posting:

https://github.com/jgyates/genmon/wiki/3.6---Serial-Troubleshooting

If you are having other issues, please see the following page:

https://github.com/jgyates/genmon/wiki/3.5---General-Troubleshooting

If you need to send you logs and registers to the developer, if you email is setup and working properly you can click send your logs on the About page in the web interface.

Expected Behavior

I've assembled and installed a RPi ZERO W based Genmon device described below on my Generac Synergy 20KW generator. I've connected to and powered it from the Evolution controller (5V). It worked for varying intervals and results until I upgraded Genmon version to V1.12.32. With the exception of the repetitive and persistent serial packet timeout errors described below, the monitor seems to be working well. -Platform: RPi Zero W Rev. 1.1, MAX3232 RS232-TTL interface powered from RPi 3.3V -Raspian Ver. 9 OS (stretch)

Actual Behavior

From the first attachment of my RPi Zero Genmon device to the generator, it has exhibited 50% serial packet losses from the Modbus - it takes two master packet transmissions to receive one slave response. An error warning is usually flashed for 3-5 seconds at varying intervals and Monitor Health alternates between OK and Not Receiving Data too. The device has run without crashing or other errors appearing for two 24 hour periods.

The HTTP monitor was active during both of these tests. I don't know if dropping the HTTP connection caused earlier crashes (before the latest upgrades). However, I noted Genmon Serial and Modbus log files were very large (up to 5 copies) during my earlier testing experiments. Perhaps these large files crashed the system.

Steps to Reproduce (including precondition)

I suspect that configuration of the RPi Zero serial port may be at fault. It passes all loop-back testing. Perhaps there is interference with another application or driver running in the full version of Raspbian 9 being used. This OS is now a clean install from NOOBS 3.0.1/ No other applications have been installed or activated. Is it possible that that the serial port is improperly configured for the ttyAMA0 on the RPi ZERO ? Since the serial port/Modbus packet losses are persistent, I think resolving it is a priority.

Screenshot or Pictures relating to the problem (if possible)

{Please write here}

Your Environment

• Generator Model: Generac Synergy (NG)
• Generator Registers: Submit via the About page
• Genmon Version: V1.12.32

[liltux]

So this packet loss only started with the software upgrade? I only ask because I have a Pizero running genmon on a friends generator and may need to go on site to check. As far as I know his is working well and I receive the weekly messages when it exercises.

[erfahnoe]

No, the repetitive packet loss issue (master transmits 2 packets- receives 1 packet with no other errors) has been there since the original installation of Genmon V1.12.10 and the NOOBS rev 3.0.0 version of Raspian 9. The upgrade to V1.12.32 seems to help with the crash issue but that has not been tested thoroughly. What version of Raspian did you use on your friends Genmon? Does it have an Evolution controller? I have enabled the slow CPU and disable power display options. The CPU utilization is normally in the low 20% range now.

[liltux]

I don’t remember any packet loss at the time of installation. It would have been raspbian lite from last summer as it’s been installed and running since last August sometime. I will have to ask him to take a screenshot and send it to me.

[jgyates]

@erfahnoe

When you say "master transmits 2 packets- receives 1 packet with no other errors", do you mean that there is a timeout error but no other errors? The software should log a timeout error at a minimum before sending a second packet.

I would guess that your problem is one of the following:

1) there is an intermittent problem with your cable. Do you notice the problem getting better or worse if the generator is running? This may indicate a marginal connection in your cable. The vibrations from the engine running affecting the comms may be an indicator of a problem connection. 2) Your ground connection is not making a good connection. This will also cause problem with serial communications. 3) You are having a power problem. Can you send the info from the Monitor page so we can see the serial stats and pi undervoltage info?

Also, if you have outbound mail working you can send me your logs on the About page. This may give a hint as to what is happening.

Jason

[liltux]

I agree with Jason, sounds like a cabling issue, ground, or even a flaky ttl converter. I just checked a screenshot of my friends and no issues. It is a pizerow running power from controller and serial.

[erfahnoe]

[erfahnoe]

Sorry about the delay. A recent screenshot of of Monitor is above. Will try to send log files.

[erfahnoe]

Sorry, previous screenshot was of an inactive Genmon device. Here is the active one I've been discussing!

Screenshot_2019-04-17 Generator Monitor

[jgyates]

@erfahnoe

I got your second screenshot.

I received you log files. I actually received two sets of logs. Which set is the one you are having a problem with?

In the first set of logs I recieved, it looks as if the serial of TCP option is enabled. The log file /var/log/myserialtcp.log shows a TCP connection could not be established. This log file would not be present if TCP were not enabled. The first set of logs also appear to have some invalid settings in the /etc/genmon.conf file, specifically the additional_modbus_timeout. This is editable via the advanced page.

The second set of logs does not include the serial over TCP log file but the serial log file (/var/log/myserial.log) shows errors that indicate the serial port is not setup properly.

Error: /dev/serial0:device reports readiness to read but returned no data (device disconnected or multiple access on port?) : myserial.py:103

On the pi is would run this command

1) Log into your pi via ssh 2) Change directories into the genmon folder

cd genmon 

3) Change directories into the OtherApps folder

cd OtherApps 

4) Run this command:

  sudo python serialconfig.py -c

The output should look like this:

 Checking : BT service disabled: OK
 Checking : Serial console command line removed: OK
 Checking : Is enable UART in boot config: OK
 Checking : Is BT overlay disabled: OK
 Checking : Serial console service disabled: OK

Serial port settings are OK. A reboot is needed if changes were made.

If any of these are not OK. Then run this command:

sudo python serialconfig.py -e

Then reboot.

If you see everything as OK then I would recommend copying a fresh copy of genmon.conf to the /etc directory and setup again using only the web interface. You mail settings should remain as they are contained in /etc/mymail.conf.

If you are logged into your pi and in the genmon folder you can reset all of your settings to the default by typing:

  sudo cp /etc/genmon.conf /etc/genmon.conf.bak
  sudo cp ./conf/genmon.conf /etc 

Then reboot.

[erfahnoe]

The generator has not been running since this Genmon monitor was reinstalled and powered up 22 hours ago. I have seen no evidence of intermittent connections to the Evolution controller. All serial loop back testing has been done at the Evolution connector and powered by a plugin 5V power supply with no faults detected. I have not measured the Evolution 5V output with the monitor connected but it was at 5.02VDC on several occasions when disconnected.

[erfahnoe]

I'm not Linux savvy. If you want me to login to the active RPi device remotely with ssh, I don't know how to do that. I have been checking and configuring this device by removing it from the generator and connecting it to a monitor and keyboard/mouse. Is this a problem?

[erfahnoe]

The active Genmon device is the second screenshot and set of logs I sent you above.

[jgyates]

You can make the changes I mentioned above with a keyboard and mouse. It just may be easier on you to use ssh. This will keep you from having to disconnect and reconnect every time you change something.

If you want to use ssh you need to know the IP address of the PI? Also, what operating system are you using on your PC (Windows or Mac). If you are using Windows you can use the program PuTTY. This pages tells how to use it an where to download it.

https://mediatemple.net/community/products/dv/204404604/using-ssh-in-putty-

If you use a Mac, it comes with ssh. Just open a terminal and type:

 ssh pi@IPADDRESS

where IPADDRESS is the IP address of your PI. It will prompt you for the pi password. Once you type the password you will be logged into the raspberry pi as the pi user in the pi user home directory where genmon should be installed.

[erfahnoe]

I tried using ssh pi@192.168.1.x before, the connection to port 22 was refused. Should another port be specified or does the RPi have to be set to accept port 22?

[erfahnoe]

Figured out that ssh has to be enabled. Thanks for being patient with this bumbling newbie. Will follow Jason's instructions and get back today

[erfahnoe]

Found all serial.config settings were OK and reset genmon.conf as instructed. Also enabled ssh, it works and I can login. Reinstalled the Genmon device in the generator and powered it up. I was apparent immediately the original 1 out of 2 packet loss issue persists. Reconfigured the Settings as before and saved them. The packet loss issue continues. Initialization takes a while with the packet losses and this save/reboot time did not identify the Evolution controller. Here is a Monitor screenshot: Let me know if you have any suggestions for further checking or changes in the Raspbian or Genmon configuration. I can setup port forwarding on my router if you would like to access the Genmon device directly. Thanks for your help.

Guess I will check the ground and 5VDC connections to the generator next.

[liltux]

By the way, is this Evo1 or Evo2? Evolution controller 1.0 or 2.0

[erfahnoe]

I measured operating DC and AC voltages at RPi pins 1, 2, 8, and 10 versus common (pin 6) as follows:

Pin 1 = 3.29-3.30 VDC, 0.000 VAC Pin 2 = 4.93 +/- 0.01 VDC, 0.030-0.005 VAC Pin 8 (TX) = 3.30-3.20 VDC, 0.000-0.18 VAC episodic as expected Pin 10 (RX) = 3.26-3.21 VDC, 0.000-0.11 VAC episodic as expected

I also measured the the voltages between common (pin 6) and the generator chassis (not the battery negative terminal) as 0.000 VDC and 0.028 VAC steady. I saw no significant dips in any of these readings. However the generator was not running.

From this, I conclude the likelihood of a problem with the wired connection to the Evolution controller is very low. The MAX3232 RS232-TTL interface has not been electronically tested. However the master/slave packet ratio 0f 2:1 at every slave response strongly indicates to me that it is not the problem.

Do you have suggestions on how we might look for OS or Genmon bugs that might impact the Genmon-EvolutionNexus serial connection this consistently? I note that Genmon does not reliably detect the Synergy Evolution controller. Once it cannot identify the controller during initialization it stays with Nexus but the 2:1 packet issue is the same. The Evolution controller only appears during some initializations. I have used Setting saves to reinitialize to get the Synergy Evolution controller identified properly in Monitor. However, the 2:1 packet issue remains. Might the Synergy identification and setup be causing the problem? I searched the closed Issues board but didn't see any Synergy related items.

I have a dual channel scope for a laptop that I can use to check the TX and RX signal voltages as well as check the RS232 signal rates and framing.

[jgyates]

@erfahnoe

My guess is that this is likely not a synergy specific problem, but rather a serial specific issue or possibly a faulty controller. If the software is missing half of the packets requested then that could cause the software to identify the controller incorrectly. To identify the model for Evolution / Nexus register 0000 is used. If this register is a 0x000a then it is a Synergy model. If the modbus read for register 0000 times out then the value is undefined. You can view the registers read from the controller on the register page by clicking on the icon in the upper right of the web interface.

A way to rest just the serial link is to use the modbusdump.py program. To use this log in via ssh then type:

cd genmon
bash ./startgenmon.sh stop
cd OtherApps
sudo python modbusdump.py -r 9600 -p /dev/serial0 -a 9d -s 0 -e 2

This will read the first 3 modbus registers. The output will look like this.

Baud Rate : 9600
('Port is :', '/dev/serial0')
Address is : 9d
Start Register : 0
Start Register : 2
0000:000c
0001:0000
OK

If you are missing these lines:

0000:000c
0001:0000

Then there was no response from the controller.

To start genmon again you can either reboot:

  sudo reboot 

or change directories back to the genmon directory:

cd ..

And restart genmon:

bash ./startgenmon.sh restart

Also, thing to do is to send pictures of your connections from the pi to the converter and cable. If your converted powered by 3.3V or 5V? I have seen both work but 3.3V is recommended. This page has more info : https://github.com/jgyates/genmon/wiki/3.1--Making-a-Cable

To attached pictures you will need to use the github web interface from a pc. The mobile app and replying to github emails does not support attaching files.

[erfahnoe]

Ran you modbusdump test, here is the result:

pi@raspberrypi:~/genmon/OtherApps $ sudo python modbusdump.py -r 9600 -p /dev/serial0 -a 9d -s 0 -e 2 Baud Rate : 9600 ('Port is :', '/dev/serial0') Address is : 9d Start Register : 0 Start Register : 2 0001:0000 OK Looks like only the 2nd register was received.

The converter is powered by RPi pin 1 - 3.3 V. I'm in the process of building a new baseboard for RPi Zero and will use a new converter. Maybe I damaged it early on when I inadvertently connected a 5v supply to RPi pin 1 and fried the first RPi Zero. Also want to rearrange the baseboard a bit and add a on-off switch.

Rebooting.

[erfahnoe]

I'm using an older Intel PC running Ubuntu Linux (18 LTS) to work on this project. It's too complicated and confusing to work with Linux in Windows.

[jgyates]

replacing the pi and / or the converter may be worth a shot. You can also try modbusdump.py with a longer sequence:

sudo python modbusdump.py -r 9600 -p /dev/serial0 -a 9d -s 0 -e 9

this will read the first 8 registers. This could tell if it is random or every other one.

[lmamakos]

I recall when I built my RPi Zero in a small box and I was hunting around on ebay for the TTL/RS-232 level converter module... there were some substandard/counterfeit versions available. Not suggesting that's exactly the problem here, but maybe don't discount the notion that the level converter could be flaky. You might have even lower margins running at 3.3v on those converters vs. 5 volts, even though its supposed to be "in spec".

[erfahnoe]

Jason; I will run the longer modbusdump,py and send you the results shortly. The last reboot has been running in Nexus mode for 12+ hours. The 2:1 Modbus packet loss pattern is the same.

It occurred to me that my Synergy was installed with a 3G wireless Mobil Link. The usefulness of this accessory wasn't worth the the price of renewing the annual service subscription. I removed the Mobile Link and it's harness to plug in the Genmon monitor. I believe the Evolution modbus port is used to support other Generac accessories. I suspect the modbus port may be configured specifically for the 3G wireless Mobil Link. I think use of cellular must require configuration and steps to make the cellular connection. What do you think? Have you run into this issue before? The Generac manuals I have only mention two Dealer programmable timer settings but I know there are many more. I asked my Dealer to change the Start-up Delay Timer when the generator was serviced last Spring. He showed me how it was done. Do you have any information about Modbus related settings that can only be adjusted by a Dealer?

[jgyates]

You should be able to access the dealer menu with the instructions on this page:

https://github.com/jgyates/genmon/wiki/Appendix-2--Generator-Controller-Information-(Registers,-Protocol,-etc.)

I am not aware of any settings on the controller for mobile link. I used to have 3G mobile link too. However it is possible they have changed something in the Synergy Firmware. I am also not aware of any settings on the controller the change the way modbus works on the controller.

One thing you could try: I have seen some general flakeyness with newer version of the firmware (Evolution 2.0) as it relates to serial communications. In once instance the person was having a problem and a power cycle of the controller fixed the issue. They unplugged the connectors on the back of the controller and reconnected them to power cycle the controller. From my understanding your settings should remain in place. I have power cycled my controller (Evolution 1.0 Liquid Cooled) with no issues.

[erfahnoe]

Here is the result of the latest modbus dump: pi@raspberrypi:~/genmon/OtherApps $ sudo python modbusdump.py -r 9600 -p /dev/serial0 -a 9d -s 0 -e 9 Baud Rate : 9600 ('Port is :', '/dev/serial0') Address is : 9d Start Register : 0 Start Register : 9 0000:000a 0002:0000 0004:0000 0006:0001 0008:0000 OK Exception in thread SerialReadThread (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner File "/usr/lib/python2.7/threading.py", line 754, in run File "/home/pi/genmon/genmonlib/myserial.py", line 103, in ReadThread File "/home/pi/genmon/genmonlib/myserial.py", line 158, in Read File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 506, in read <type 'exceptions.TypeError'>: 'NoneType' object is not callable I see only 5 responses.

I will try restarting the Evolution 2 controller. I think this can be done by turning the generator OFF, disconnecting the 7.5A fuse and disconnecting the power to the battery charger and reversing the process. This should reboot the Genmon RPi too.

[liltux]

Just to clarify for confusion sake. There are three different Evolution controllers: Evo 1.0 Evo 2.0 Synergy Each of these controllers are different and have explicitly different firmwares and wiring conventions. All three are compatible with mobile link.

[jgyates]

That is interesting. In the first run of modbusdump.py you missed register 0000 but you got it in the second so it look like every other packet sent is ignored by the controller.

@liltux is right to point out that Evo2 is different from Synergy. My point, which I did not make very clear, was that both Evo2 and Synergy are relatively new variants in comparison to Evo1.0 and may contain some bugs in the firmware that a controller power cycle may help.

[erfahnoe]

Restarting the Evolution controller with the RPi powered from the controller 5V pin doesn't solve the missed packet issue. Maybe that's the reason your basic RPi 3 design works by powering the Genmon RPi directly from the battery - Genmon is already running when the Evolution controller starts up. Is this worth trying?

[jgyates]

I guess at this point anything is worth trying :)

If that does work I would guess that it would be about the power more than the timing of genmon starting since the controller is the slave and will not output anything unless genmon request a modbus register.

[erfahnoe]

I'm not surprised that Synergy controller firmware is unique as @liltux states. It is so unique that it has been discontinued for now. I'm told by my dealer that the Synergy experienced some issues with some of the hardware - the throttle stepping motor was mentioned. I suspect frequently cycled heavy loads like AC units are a problem. There is a rumor the Synergy technology will be re-introduced at some point. At the moment, I plan to turn off the AC when we're on vacation and not use the dryer or AC whenever a prolonged power outage occurs. My dealer told me that the WiFi version of Mobil Link is only available on new factory units and there is no rollout plan for retrofitting old generators at the moment. The service technician said that both the Evolution and Synergy controller firmware is likely to be upgraded to support WiFi Mobil Link and that the updated controller may be more difficult to hack afterward. Sounds like a challenge. For my part, I'm happy with with the functionality of Genmon in Nexus mode though the AUTO generator/switch control available with the Evolution/Synergy mode would be useful. I'd just like to get rid of the modbus packet losses and their persistent warnings and error logging every other cycle. I can live with the warnings but I fear that logging the serial and modbus errors will cause problems. Can this logging be turned off? I really just need the email notification and remote generator control features, the Status and and Monitor screens are useful too. If the WiFi version of Mobil Link becomes available for my Synergy, I will consider it.

[jgyates]

One thing you can do to preserve the life of your SD card due to the logging is to make all of the logs go to a ram disk. This article gives instructions on how to do that:

https://mcuoneclipse.com/2019/04/01/log2ram-extending-sd-card-lifetime-for-raspberry-pi-lorawan-gateway/

another thing you can try is to override the controller detection. To do this edit the /etc/genmon.conf file:

sudo nano /etc/genmon.conf

scroll down to around line 96 and change this line:

#liquidcooled = True

to this:

liquidcooled = False

Then around line 102 change this:

#evolutioncontroller = True

to this:

evolutioncontroller = True

To save the file type Ctrl+x then type 'y' for yes and hit enter. Then restart. This will force the software to assume an evolution air cooled controller. This may work better than the fall back to Nexus mode.

[erfahnoe]

Tried running my Genmon device when powered with a 5V micro USB supply. (Disconnected only the 5v source from the controller connector by removing a fuse, the Common, Tx and Rx connections were in place.) I energized the USB supply and Genmon started up in Nexus mode as usual with the packet loss errors. Is used a trick that has worked before (changing a saving a stetting) to reinitialize Genmon into Synergy mode. Packet loss errors resumed as usual. Then I rebooted the Synergy controller (remove 7.5A fuse and disconnect battery charger and reverse the procedure). Unfortunately, the packet loss errors came back, see the attached screenshot:

Notice that the packet loss percentage is slightly above above 50%, this small master- slave offset (9 extra master transmissions)appeared at the beginning after the packet totals were reset. Note also that the Synergy mode was retained. I then reinitialized Genmon by changing the fuel selection to propane and got the Nexus mode back. After a few fuel switches (propane to NG and vice versa) I was able to get the Synergy mode back: I had hopes that the Synergy mode supported putting the controller back into the Auto mode in Maintenance but it only adds a Start and Transfer button.

Since we can't make the modbus packet loss issue go away by powering up the Genmon before restarting the Synergy controller, I'm going to restore the restore the Genmon power supply to the controller 5V source. Do you think changing any of the Modbus Advanced settings would be helpful?

[jgyates]

If the software reads modbus register 0000 correctly it will identify the controller correctly. If it does not it will assume nexus.

You could try setting Additional Modbus Timeout (in seconds) on the advance page (click the icon in the upper right twice) to 3. This will add 3 additional seconds to the current timeout.

[jgyates]

FYI, the only times I have had to use the additional modbus timeout is when serial of TCP/IP option is used, depending on the network response. Once I connected to a remote generator and this option was needed, but it may help in your situation.

[erfahnoe]

Jason; Added 3 seconds to Modbus Timeout. Didn't help, just slows things down. First save resulted in Nexus. Fuel switch to Propane resulted in Synergy controller after reinitializing but no change with packet loss. I reverted back to 0 added seconds, it reinitialized to Synergy this time. I suspect it matters what state the Monitor Health is in when the Save initialization begins. but identifying Synergy at initalizattion doesn't fix the Modbus packet loss issue it just syncs the register 0000 issue.

I will finish my new RPi Zero backboard and see if the new MAX3232 fixes things. Will let you know. Jason, you have been very helpful and attentive to my difficulties, THANK YOU!. I also appreciate the efforts of all the other commentators.

[liltux]

@jgyates I had a few minutes this afternoon and plugged up a pizerow to a synergy controller in our shop, no serial communication at all. This unit was working on a Evo1.0 controller, just removed and installed on this shop generator. I will look into my installation better over the next week. Is there anyone else with a Synergy Generator on here that could also add? My curiousity, on the Synergy units, the AVR (Automatic voltage regulator) is external to the controller. When the Synergy AVR is active (powered up during run and cool down cycles) there is constant serial communication between the two. I am wondering if there is something there between the 2 interfaces. I am going to try a pi3b+ as well, will let you know if different.

[liltux]

@jgyates ok, i believe my pizero w is not 100%. I connected it to an Evo1.0 and it is communicating but experiencing packet loss. Good, note. I plugged in a Pi3B+ on the Synergy powering from battery and it is communicating without any loss. Just FYI.

[jgyates]

@liltux , these are great data points. I wonder if the pi zero kernel has some type of power management that the pi3 does not and this is causing communications issues.

[jgyates]

@liltux , Was your pi3 using the same serial converter as the zero? I am wondering if the pi zero has a problem supplying power to the converter. I know that may be a long shot but just trying to rule things in or out of the equation.

Another thing to try would be a USB serial adapter. If the results are the same then we could rule out the serial converter.

[liltux]

@jgyates both devices were using similar serial adapter but I was not using the same one on each. I was fixing one for a friend and just happened to have both handy at the moment.

[erfahnoe]

Problem Solved! Installed my upgraded base board for the RPi Zero, connected it to the Synergy generator- Bingo no serial0-Modbus packet losses. Here is a screenshot: https://screenshots.firefox.com/nlE01cJm5a7B5xbF/192.168.1.9

The persistent 50% RS232-Modbus packet loss problem was due to the RS232-TTL converter on my original base board. I used a new converter from the same lot on the new base board. I did some research on the MAX3232 chip and noted that it has a voltage regulator on the charge-pumps for the RS232 interface and that operation on 3.3V reduces the performance if the TX output is heavily loaded. This can be mitigated somewhat by increasing the charge-pump capacitors from the 0.1uF standard. Like @liltux, I was also concerned about the capacity and stiffness of the 3.3V supply (pin 1) on the RPi Zero. Hence, the MAX3232 converter is now powered from the 5V supply (pins 2&4) on the new base board. I added a 2.7K resistor between the RX output on the MAX3232 and the RX input of the RPi Zero. This should keep the current into the RPi RX input well below maximum specification of the RPi during both startup and operation. It certainly works well a 9600 baud. I have not yet tested the MAX3232 on my original base board but will change that board to drive the chip with 5V and see how Genmon works. I'll let you know. In any event, I think using the 5V supply for the RS232-TTL converter when using a RPi Zero W.

Jason,thanks again for your great support, I look forward to using Genmon.

[jgyates]

@erfahnoe

thanks for the feedback. I am glad it is working for you.