Question on Max Line Voltage

[hckrdan]

Hey Jason, thanks for all the great work on the genmon project! I sent you a tip for a coffee! Question: my genmon connected to a pre-Nexus controller on a Generac m.n. 55040 shows a Max Utility line Voltage of over 40,000 volts, where is that coming from?

Line : Utility Voltage : 242 V Utility Max Voltage : 40475 V Utility Min Voltage : 207 V Utility Threshold Voltage : 133

[jgyates]

Hi @hckrdan

The Utility Volts Min and Max are both tracked in the outage monitoring thread. The software reads the utility voltage and just stores the values. This out of range value could mean that you have communication issues (timeout errors). If the controller does not respond within a fixed period of time (about 3 seconds) then the software moves on to the next packet. If the controller does respond once genmon starts the next request this can lead to invalid data (like the 40475 value). Look on the Monitor page to see if there have been any communication errors. You can restart the software and this value should change back to a reasonable value, but keep an eye out for how often you are having communication issues. If it is only once in a long time then I would not worry about it. You can extend the modbus timeout on the advanced settings page (double click the gear icon to get to the advanced settings page). If you get a lot of these then you may want to validate your cabling.

Let me know if that does not make sense or if you have any other questions.

[hckrdan]

Thanks Jason, makes sense although I don't see a huge number of errors, I'll try to restart genmon, I didn't think pre-Nexus use modbus protocol:

Monitor :

Generator Monitor Stats :
    Monitor Health : OK
    Controller : Pre-Nexus, Air Cooled
    Run time : Generator Monitor running for 20 days, 17:50:20.
    Generator Monitor Version : V1.18.14
    Update Available : No

Communication Stats :
    Packet Count : M: 5720733, S: 5719271
    CRC Errors : 0
    CRC Percent Errors : 0.00%
    Timeout Errors : 1461
    Timeout Percent Errors : 0.03%
    Modbus Exceptions : 153
    Validation Errors : 52
    Sync Errors : 52
    Invalid Data : 111
    Discarded Bytes : 0
    Comm Restarts : 0
    Packets Per Second : 6.38
    Average Transaction Time : 0.3107 sec

Other weird thing I see is multiple 3 second outages in the outage log (but there were no power outages as far as I know): Outage : Status : Last outage occurred at 2022-10-04 20:26:09 and lasted 0:00:03. System In Outage : No Utility Voltage : 242 V Utility Voltage Minimum : 207 V Utility Voltage Maximum : 40475 V Utility Threshold Voltage : 133 V

Outage Log : 
    2022-10-04 20:26:09, Duration: 0:00:03
    2022-10-04 20:26:02, Duration: 0:00:03
    2022-10-04 20:25:55, Duration: 0:00:03
    2022-10-04 20:25:48, Duration: 0:00:03
    2022-10-04 20:25:42, Duration: 0:00:03
    2022-10-04 20:25:36, Duration: 0:00:03
    2022-10-04 20:25:30, Duration: 0:00:03
    2022-10-04 20:25:23, Duration: 0:00:03
    2022-09-27 17:43:48, Duration: 0:00:03
    2022-09-25 02:47:56, Duration: 0:00:03
    2022-09-21 01:49:18, Duration: 0:00:03
    2022-09-18 00:44:40, Duration: 0:00:03

[jgyates]

Yep, that is a lot of timeout errors. Zero is typical.

You have a comms problem. You can troubleshoot it, or you could put a band aid on it by adjusting these settings on the advanced page:

Minimum Outage Duration - outage must be x seconds (put around 10 sec) Additional Modbus Timeout (sec) - Try putting 5 or 10 in this setting

this will make it more tollerant to timeouts but you still have an underlying issue.

Yes. pre-nexus does use modbus, the addressing is a little different.

[hckrdan]

Thanks again Jason! For some reason I don't have the advanced settings page - maybe because I am using a serial to TCP/Wi-Fi adapter? (forgot to mention that initially). I'll check if there are any settings in the adapter. The voltage reset to a reasonable value but it jumped to 40kV again after a while.

[jgyates]

Double click the gear icon in the upper right corner of the web interface to access the advanced settings page. Tcp/wifi does not matter.

[hckrdan]

Got it, thank you, I changed those settings and will let you know if problem persists.

[jgyates]

Since you are using serial over TCP via wifi, your communication issues could be related to your wifi signal. wifi is not a perfect medium and some errors are expected. I use serial over wifi and as a refernece here are my comm stats at the moment:

Communication Stats : Packet Count : M: 25898217, S: 25898202 CRC Errors : 0 CRC Percent Errors : 0.00% Timeout Errors : 14 Timeout Percent Errors : 0.00% Modbus Exceptions : 0 Validation Errors : 5 Sync Errors : 5 Invalid Data : 1 Discarded Bytes : 0 Comm Restarts : 0 Packets Per Second : 30.60 Average Transaction Time : 0.0646 sec

I use additional modbus timeout of 10 and a minimum outage duration of 4

ok. I am going to close this issue for now. But feel free to post here with follow up questions or status.

[hckrdan]

Thank you again Jason, I set both values to 7 but issues still persist, still shows 40475V, I'll try with 10, though you're right about the WiFi, I do get periodic text messages via Twillio that the system is "not receiving data" but errors are 0.01%

Monitor :

Generator Monitor Stats :
    Monitor Health : OK
    Controller : Pre-Nexus, Air Cooled
    Run time : Generator Monitor running for 23:21:56.
    Generator Monitor Version : V1.18.14
    Update Available : No

Communication Stats :
    Packet Count : M: 265095, S: 265055
    CRC Errors : 0
    CRC Percent Errors : 0.00%
    Timeout Errors : 39
    Timeout Percent Errors : 0.01%
    Modbus Exceptions : 2
    Validation Errors : 0
    Sync Errors : 0
    Invalid Data : 5
    Discarded Bytes : 0
    Comm Restarts : 0
    Packets Per Second : 6.30
    Average Transaction Time : 0.3138 sec

PS: I set the serial speed on the Serial to WiFi adapter to 9600 bps, I think that was the right setting recommended in the docs, I do see your packets per second rate is much higher

[jgyates]

your packets per seconds is pretty bad for wifi. This tells me that there are a lot of wifi retries going on. Every packet sent out on wifi gets a reply acknowledge (ACK) packet from the access point. There is a timeout for waiting for the ACK and a predefined number or retries to send the packet if it does not receive and ACK. This process is not in sync and is taking longer than the modbus timeout which is the source of your problems.

You may want to try a larger antenna on your serial to wifi adapter. I use this one:

https://www.pusr.com/products/1-ports-wifi-to-serial-converters-usr-w630.html

and you can replace the antenna. https://www.data-alliance.net/ is a great source for wifi antennas and extension cables for antennas.

Example of a large antenna

https://github.com/jgyates/genmon/blob/master/Diagrams/Antenna.jpg

[hckrdan]

I have a very similar serial-to-WiFi adapter, the USR-W610 from the same company and also have a long range Hawking HAO14SDP hi-gain antenna on it. Latency from Pi to Serial -2-WiFi is between 2 ms and 9 ms ping time from the Pi. For now seems to keep the Max voltage to reasonable value after I raised modbus delay to 10s, I'll keep an eye on it and keep you posted, maybe the latency is due to the older controller. Another unrelated think I wanted to ask is whether genmon monitors the two contact sensors my engine has for oil temp and pressure, if not, I can provide the registry and bit numbers for them (from the service manual)

[jgyates]

No, genmon does not currently monitor this info. If you have info on modbus registers I could add this.

[hckrdan]

Ah, I don't have that, just the bits from digital input registers one can access via the Debug menu (is that register 1 from Live Register View?): Position Digital Inputs Digital Outputs 1 Low Oil Pressure Not Used 2 High Temperature Not Used 3 Internal Use Not Used 4 Internal Use Not Used 5 Internal Use Fuel 6 Not Used. Starter 7 Auto. Ignition 8 Manual Transfer

Thank you for your work and your help, you're the best!

PS: checked my WiFi router and the connection to the USR shows 72 Mbs TX and 65 Mbs for Rx, so pretty decent connection.

[jgyates]

Genmon uses these bits for Evolution, however the Nexus (and Pre-Nexus) do not appear to export this data over modbus. This is exported via modbus registers 52 (inputs) and 53 (outputs). Modbus registers 1 is the engine and switch status registers.

[jgyates]

This may not make a difference but another thing I do is I put my pi on a wired connection. This takes out some of the latency with two wifi connections. With both on wife, one modbus transaction would go from the pi over wifi to the access point then from the access point to the USR converter then the response back would go in the reverse path. This is TCP so each packet is ACKed and the underlying wifi packet are also ACKed so there is a lot of room for timeouts in a noisy environment. Putting the pi on a wired connection may help.

[hckrdan]

That makes sense but my Pi is in a different location/floor than the router because I'm also using it to display the hourly electricity price and is a bit more difficult to run a wire between locations. However, I noticed something else: the max voltage displayed (40475) is the same value (0x9e1b) that a bunch of other unused registers (69 of them to be precise from register 002d to register 05fa) are showing, so I think genmon reads the value from one of those non-existent "registers" not from the 0009 Utility Voltage register - which doesn't have the 40475 value in the 10min/1 hr/24 hr graph of values that it recorded (see attached pic) I will try with the hardwired link this weekend but I'm a bit skeptical it would solve the issue.

[jgyates]

yes, those registers are not used for pre-nexus. They are used by either Nexus or Evolution. The way genmon is structured is that it reads a fixed list of modbus registers and stores them. It repeats this reading over and over in a single thread. Other threads interpret and display info form the stored registers. Because of what is happening with network timeouts the software is reading the utility voltage register and it is timing out so the software moves on to read the nest register in the list, but the controller responds late enough for it to be confused with the previous request. Even if the software did not read the non pre-nexus registers the value would still be incorrect as it would belong to another modbus register. The modbus protocol does not support canceling a read transaction and there is not time out period defined by the protocol so there is not a way to correct this except to try to validate the data as much as you can and try to make your communication medium as reliable as possible.

Genmon read register x Genmon waits for a response for register x Genmon timeout waiting for register x response Genon read register y Genmon waits for a response for register y Controller response with register x value Genmon interprets this response a a value for register y (not x)

In the above list of events register y is the register with the utility line voltage. In an installation with robust communication medium a given register (like the utility voltage register) is updated about every 3 -5 seconds assuming no timeouts. Any errors like the one above only last for a short period of time (hence the outages of 3 seconds). I wish modbus would support canceling of requests or even a way to number packets, but unfortunately it is a very simple protocol and network signal problems with wifi really expose the limits in the protocol. The hardwired may not solve the issue, but if your wifi to the pi is marginal then it will help.

[hckrdan]

Thanks Jason, makes sense. I'll try to wire the Pi temporarily to see how that works and keep you posted.

PS: is there a way to reset alarms (like "check battery") from Genmon UI? (As it's raining now and just got an alarm 🤣)

[jgyates]

Not for nexus, pre-nexus, or Evo Air cooled. Only the Evolution Liquid cooled model supports this functionality.

[hckrdan]

Can we place an upper limit on the Max Voltage value such that if genmon reads one of the other registry values that is way outside the reasonable voltages (i.e. over 270V) it just ignores it? Where would I do that in the code?

[jgyates]

I had a check like this for Evolution 2.0 as there is a know issue where the controller sometimes returns random data. See known issues number 11 https://github.com/jgyates/genmon/wiki/Appendix-D-Known-Issues

The issue you are seeing is known issues number 16

I moved the check that was happening for Evo 2.0 on the Utility voltage to all controllers.

The code change is here:

https://github.com/jgyates/genmon/blob/c0ff004a9e16ce80d94ddf6bc489b37df4e741f6/genmonlib/generac_evolution.py#L2277

Just update your software from the about page. The version number has not changed.

[hckrdan]

Thank you Jason, looks good! I appreciate your help! Now my next challenge is to get a free SSL cert from Let's Encrypt or similar without opening port 80 on my router - I have genmon exposed via ngrok (I should be able to use a domain I own via DNS Validation)

[skipfire]

Why do you need an SSL cert if you aren't exposing it? Ngrok can do SSL without your source being SSL. As for port 80, I open up port 80 for a couple minutes when I do the Let's Encrypt setup/renewal, then I close the port again.

[hckrdan]

Hey Jon, I'm exposing it via ngrok and my domain, ngrok requires paid account for ngrok tls so I'm not using that, ngrok http also discloses your home IP so I'm using ngrok tcp which maps genmon to a high port on x.ngrok.io.

[skipfire]

I have the free ngrok working for port 80, after starting the URL works and came out like the line below and it was exposing my HTTP service with a cert. The entire subdomain has been changed from what it really reported. It isn't on my domain, but I can setup a CNAME for that. https://af72-631-101-098-44.ngrok.io -> http://localhost:80