jgyates / genmon

Generac (and other models) Generator Monitoring using a Raspberry Pi and WiFi
GNU General Public License v2.0
374 stars 76 forks source link

Serial port error #4

Closed JeremyStevenson closed 7 years ago

JeremyStevenson commented 7 years ago

jgyates - this is awesome that you've done this. THANK YOU!

I've run into a few problems on setup, and would appreciate some help.

First, I ran into the exception below. I solved that exception by putting "enable_uart=1" at the end of /boot/config.txt. Looks like this is required for all Raspberry Pi 3s if you are using the built in serial port (as opposed to a usb serial port). If this is the correct fix, you may want to add this to the documentation. I am using a Pi Zero W.

Second, when I tried to run ClientInterface.py from the OtherApps directory I got an exception that mylog.py wasn't found. I moved the program to the same directory as mylog.py and problem solved.

Third, and this is where I'm stuck, when I got ClientInterface.py running I got a warning, "WARNING: Not receiving data. : ," I've tried switching around the TX and RX lines in my wiring, and no luck. I don't get any output from genmon.py as if it is hung. Am I supposed to get output from genmon.py? Is there some basic direct access of the serial port I can do to test my hardware?

Thanks in advance, and thank you for writing this including the very detailed documentation.

-Jeremy


Exception from First issue:

pi@GeneratorRPi:/genmon $ sudo python genmon.py Traceback (most recent call last): File "genmon.py", line 2510, in MyGen = GeneratorDevice() File "genmon.py", line 334, in init self.FatalError("Error opening serial device: " + str(e1)) File "genmon.py", line 2429, in FatalError raise Exception(Message) Exception: Error opening serial device: global name 'FatalError' is not defined Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs func(*targs, *kargs) File "genmon.py", line 2434, in Close self.mail.sendEmail("Generator Monitor Stopping at " + self.SiteName, "Generator Monitor Stopping at " + self.SiteName ) AttributeError: GeneratorDevice instance has no attribute 'mail' Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs func(targs, **kargs) File "genmon.py", line 2434, in Close self.mail.sendEmail("Generator Monitor Stopping at " + self.SiteName, "Generator Monitor Stopping at " + self.SiteName ) AttributeError: GeneratorDevice instance has no attribute 'mail'

jgyates commented 7 years ago

Hi Jeremy,

Thanks for the feedback. I will add your suggestions to the documentation.

Regarding the ClientInterface.py issues you describe, ClientInterface.py takes one command line argument which is the IP address of the system running genmon.py. If not argument is provided then the default is the localhost IP address of 127.0.0.1 (i.e. ClientInterface.py is running on the same system as genmon.py). The message you are seeing (WARNING: Not receiving data) is generated by genmon.py which tells me that Clientinterface.py is establishing a socket connection to genmon.py. The warning message is generated from a thread (thread name in genmon.py is ComWatchDog()) that counts the packets received from the generator controller. The watch dog thread runs every 5 seconds so make sure that the genmon.py has been running at least 5 seconds so it can gather some data. You can just exit ClientInterface.py and restart it, if you still get the same error then we can look at communication debugging. There is some debugging built into genmon.py but it needs to be enabled from genmon.conf. To debug serial communications in genmon.py, modify the lines in genmon.conf to enable the console output and enable the monitor statistics: "displayoutput = True" and "displaymonitor = True". This will make genmon.py output to the console display, every 10 seconds, how many serial packets have been transmitted and received. This data should allow us to see what is happening in terms of serial communications. Another thing that may be useful is the daughter board for bringing out the serial communications on the Raspberry Pi should have a couple of LEDs on it that flash when data is transmitted or received. This is another indicator that would be helpful in debugging. One note that also helped me, when making the cable I had a learning curve on crimping the molex connectors. To validate my cable was working is created a simple program that would transmit data and print what it received. I looped back the tx to the rx on the end of my cable so I was transmitting through the cable and receiving through the cable. I am attaching a simple test app that I used to test serial communications. It uses /dev/serial0 so, depending on your RPi version the name of the device may change. As I write this I recall that you may want to validate your device name of your serial device. If memory serves, they renamed the raspberry pi on board serial device name from /dev/ttyAMA0 to /dev/serial0 (assuming you are using raspbian). I can't remember if this name change started with the RPi2 or RPi3. You can check this by doing a "ls -la /dev/" from the console and see if you have a device named /dev/ttyAMA0 or /dev/serial0. SerialTest.py.zip

Let me know if how it works for you and if you have any other questions. Thanks Jason

jgyates commented 7 years ago

One other thing now that I think of it, you can also use minicom. It is a console based terminal. Once you tell it what serial port to use it will send what you type and display what is received so if you use a loopback you can test your serial port. Thanks again for the feedback. I added some quick notes on your comments to the wiki and readme. Once you get the serial port validated if you are still having issues let me know.

JeremyStevenson commented 7 years ago

Thanks for all the help. Will try this tonight or tomorrow. Also got ideas from some one I worked with that enabling hardware serial port on Pi 3 requires some additional steps. Specifically, it's linked by default to Bluetooth and needs to be un-linked, otherwise you are using a software emulating serial port. Apparently the software emulating serial port can be unreliable, especially at higher speeds. You also need to turn off the console, which is hooked up to that serial port.

Those commands are as followed (haven't tried yet):

Enable serial port

Disable serial console

You'll hear back from me soon...

jgyates commented 7 years ago

I posed and update to genmon.py that should make the output more succinct if an error occurs when the serial port is not present or busy. I also added some documentation on the genmon.conf parameters that I mentioned above. Yesterday I added some details to the docs on serial port setup.

JeremyStevenson commented 7 years ago

Ok, two steps forward, one step back.

I'm running Jessie Lite by the way.

I ran the commands from my last message and that got the serial port working. YAY! For good measure after I confirmed a working serial0 I also added "dtoverlay=pi3-dable-bt" to /boot/config.txt since I'm not using USB and I wanted the Pi to use the hardware UART.

However, now it appears I'm getting somewhat random information from my generator through genmon. One example startup I got the following email messages: Register under test, Transfer Swtich Changed State, Critical, Transfer Switch Changed State. At this time, the generator wasn't doing anything.

Thinking about that more, perhaps this normal and genmon is just sending me past log events? When I did a manual start of the generator I got the following: Generator Notice (Manual, Running), transfer switch changed state (power supplied by generator), transfer switch changed state (power being supplied by the utility line), Generator Notice (auto, off-ready). So - all of that worked ok.

I have been able to set the time of the generator.

Here I think is my real problem -I have not been able to set the exercise time via ClientInterface I'm hoping to be able to do remote starting and setting the exercise time is one way of doing it. Actually, the whole point of doing this is I have a generator at a vacation home that is on the ocean. It often doesn't exercise because of the corrosion from the salt air. If I crank it a few times, it gets going then runs fine. The idea is the genmon will tell me when it fails to start, and I could tell it to re-exercise. If that doesn't work, I'll be bugging you for the experimental remote start code. Last resort I'll put a solenoid or servo that presses the start switch for me.

I also see some strange behavior when I run ClientInterface. About 30% of the time generator: status will respond "OK : Auto, Off -Ready" and then hang. I have to control-c to get communications back.

Here is a log of ClientInterface trying to set the exercise time (I replaced > with * so I could paste here):

sudo python ClientInterface.py localhost OK : Auto, Off - Ready

Exercise: Exercise Time: Saturday 08:00 Quite Mode Off Exercise Duration: 0 min

Service: Service Scheduled: Next service in 199 hours Total Run Hours: 50 Firmware : V1.43, Hardware : V1.02

Exercise: Exercise Time: Saturday 08:00 Quite Mode Off Exercise Duration: 0 min

Service: Service Scheduled: Next service in 199 hours Total Run Hours: 50 Firmware : V1.43, Hardware : V1.02

*` Here is a log of output from genmon running for about 1 minute with debugging turned on: genmon.log.txt

I hoped that I had done something wrong with the wiring and my serial communications was unreliable. However, when I did the loopback test with minicom and your SerialTest.py (thanks!) they both looked ok. Plus, I'm not getting CRC errors which I would believe would be the problem.

Perhaps my model behaves differently than your model? I am using a car USB charger connected to the battery of the generator to power my Pi.

Thanks again for your help. If you wanted SSH to my pi, I can give it to you.

JeremyStevenson commented 7 years ago

Follow up on this. I'm periodically getting this email:

Subject: Register Under Test Body:

Time: 09:42:47 AM Date: Tue 28-Mar-2017 Nothing Changed

0329:0000 0323:0000 05f9:0000 05f8:0000 033b:0000 033c:0000 05f5:0000 033a:0000 . . . (cut the file short)

jgyates commented 7 years ago

I looks like you have enabledebug=True in genmon.conf. This enables the unknown register debugging mode. I would disable this. It basically reads nearly every register periodically and sends email to you on what changed. I will take a look at the rest of the data you send. I added some descriptions of every option in genmon.conf to the documentation yesterday.

jgyates commented 7 years ago

A couple of other thoughts:

I am running Jessie Lite also. It looks like serial communications is working fine now based on the info you sent. Let's check a few things...

what is the state of the liquidcooled option in genmon.conf? If you have a radiator on your generator it should be true, otherwise false.

What is your model number? Do you have an automatic transfer switch installed with you generator?

Is your generator currently set to exercise weekly. I noticed that the exerciser time duration is zero so this may be another item that is not the same between our generators.

run the following commands in ClientInterface.py and send me the output:

This command will give the model number register, which may help. My controller was replaced so I don't have a data point on this:

generator: getregvalue=01f4 

Then run this command so we can see what is working and what is not:

generator: logs monitor outage maint status registers

My best guess at this point is that there may be more differences between the liquid cooled vs air cooled models than I could glean from the documentation. It should be easy to work around these once I get enough data, then we can look at adding some functionality to fully support all the features on your model.

When developing the program there were two ways to set the exercise time, one was straight forward and one was very convoluted (in my view). With the convoluted way I had a some issues figuring out the right bit pattern to get the exercise time and date correct. I can possibly dig up that old code and we can see if the alternate way of setting the the exercise time works, but let's validate everything that is working and not working first.

I am curious about the remote start functionality also. Once we get our base line one what is working vs not working we can dive into that.

Regarding the strange behavior of ClientInterface.py, are you using wifi and if so what is your signal strength? You can get this form the console by typing "iwconfig". It will give you info about your wireless devices. The value I am asking for looks like this:

Link Quality=45/70 Signal level=-65 dBm

If your link quality is low that could cause problems with the TCP connection if you are dropping more packets than TCP can recover from. My generator is about 100 feet from my access point and the signal is going through a brick wall so I used a USB based wifi adapter instead of the one on the Raspbery Pi 3. The USB wifi adapter has an external antenna. Don't know if that is your problem or not. I do know that I had some issues like that before but increasing my wifi signal strength appeared to fix them.

Thanks for the feedback.

jgyates commented 7 years ago

one other question: Are you running clientinterface.py from the console via ssh or on another system? The comment above regarding signal strength assumes you are running on another system. In the end, it should be as reliable as an ssh connection if you are running the program remotely. There may be some thing I can do to make that more resilient to dropped packets but I will have to think on that...

jgyates commented 7 years ago

also, you appear to have displayunknown=True. This option just displays some register values that appear to change when the generator is running. Unknown Sensor 1: 0.0 Unknown Sensor 3: 0 Unknown Sensor 4: 0

I don't know what these are but my guess is that they are used internally by the controller, for example one of them appears to be half the Frequency output.

jgyates commented 7 years ago

Jeremy,

I went back and pulled my original code that uses the alternate method of writing the exercise time. I added an option to genmon.conf to enable using the older method of writing (uselegacysetexercise = True). I tested it on my generator and it still works with a caveat. When setting the exercise time the set time can be off by one minute. I have an idea about why this is but not sure of a good solution yet. The legacy method is a bit convoluted: you have to take the day and time you want to set and calculate the number of minutes from the current time, the bits in the register are weighted in what appears to be nearly binary weight, i.e 2,4,8,16,32,64, etc, (when I try binary it does not work as well). I had a similar problem when I would set my generator via mobile link so I am not sure if this is just the way it works or if I am missing something. Anyway, this should be an improvement from the non functional feature you had before.

One thing that may be helpful. When every you are near you generator, toggle though the menu on the controller and take pictures of the settings. Be sure to include the dealer menu as it has some additional info. With this info and a full register dump it should be simple to correct any remaining register differences, for example based on the info you provided it looks like the may be an issue with the alarm log (or your generator is new enough not to have any alarm log entries) and your setup is reading a pick up voltage of zero (the voltage that the generator turns off the transfer switch). I could make the code force this value to 200V (genmon uses this to tell you when the utility voltage is turned back on) or we could look for the register for your generator. You do appear to have a meaningful value for the threshold voltage (the voltage level that lets the generator know when to turn the transfer switch on) . Anyway, I have noticed a few differences in our registers, nothing that we should not be able to correct with some debugging techniques.

jgyates commented 7 years ago

one final note for the night. I made to minor changes to genmon.py in my last update. 1) I added all of the registers that mobile link was reading ( I had taken some of the reads out since the did not appear to be used for my generator) and 2) I created default values for pickup and threshold voltages if we read zeros. This is more of a workaround than a pure fix. We still need to figure out the pickup voltage register on your system, but the two changes I made will make that easier and also it may get rid of that power line state change you were seeing until we can get a better fix.

JeremyStevenson commented 7 years ago

Sorry I haven't been able to get to this. I'll respond with everything I can remotely later today, and then pictures of my generator console (and a register log at the time) when I can get in front of the generator - hopefully tonight.

JeremyStevenson commented 7 years ago

I looks like you have enabledebug=True in genmon.conf. This enables the unknown register debugging mode. I would disable this. It basically reads nearly every register periodically and sends email to you on what changed.

Yup, that solved that problem.

I am running Jessie Lite also. It looks like serial communications is working fine now based on the info you sent.

I agree serial com seems to be fine now. I was incorrect before when I stated I was running a Pi 3. I'm running a Pi Zero W.

what is the state of the liquidcooled option in genmon.conf? If you have a radiator on your generator it should be true, otherwise false.

As far as I can tell, it's air cooled. It seems all residential Generac generators are air cooled. I set liquidcooled to false.

What is your model number?

Model number: 0058751.

Do you have an automatic transfer switch installed with you generator?

Yes

Is your generator currently set to exercise weekly. I noticed that the exerciser time duration is zero so this may be another item that is not the same between our generators.

As far as I can tell, there is no option to set exercise time.

run the following commands in ClientInterface.py and send me the output:

pi@GeneratorRPi:/genmon $ sudo python ClientInterface.py localhost CRITICAL: System in alarm! : Auto, Off - Ready -generator: getregvalue=01f4 30303037373232393837 -generator: logs monitor outage maint status registers

ClientInterface.py hangs consistently running this command. I restarted genmon.py and was able to get a result. Here it is:

ClientInterface.log.txt

Regarding the strange behavior of ClientInterface.py, are you using wifi and if so what is your signal strength? You can get this form the console by typing "iwconfig". It will give you info about your wireless devices. The value I am asking for looks like this:

I'm running ClientInterface.py locally on the same py. See localhost in the command line I ran above.

One thing that may be helpful. When every you are near you generator, toggle though the menu on the controller and take pictures of the settings.

See the google photos I shared with you.

I went back and pulled my original code that uses the alternate method of writing the exercise time. I added an option to genmon.conf to enable using the older method of writing (uselegacysetexercise = True).

No dice. See ClientInterface output:

-generator: maint Maintenance:

Exercise: Exercise Time: Saturday 08:00 Quite Mode Off Exercise Duration: 0 min

Service: Service Scheduled: Next service in 199 hours Total Run Hours: 50 Firmware : V1.43, Hardware : V1.02

-generator: setexercise=Wednesday,20:32 Set Exercise Time Command sent (using legacy write) -generator: maint Maintenance:

Exercise: Exercise Time: Saturday 08:00 Quite Mode On Exercise Duration: 0 min

Service: Service Scheduled: Next service in 199 hours Total Run Hours: 50 Firmware : V1.43, Hardware : V1.02

jgyates commented 7 years ago

I logged in to you pi and did some looking at genmon.py while running. At a minimum the following do not appear to be working as it does on my system:

Reg 0053 - Output Relay status regs (Transfer status, Battery Charging status, etc) Reg 0057 - Input status registers (used for telling the type of alarm) Reg 023e - Exercise time duration (not essential, but nice to display) Reg 023b - Pick Up Voltage (not essential, but nice to have)

I have a couple of other things that are not looking right but I need to validate this with your photos (alarm code / alarm log and maintenance log don't look right but you may have an empty log and I am just not processing it correctly. The start / stop log looks great, which is a bit confusing since they are mostly formatted the same way.

I made some minor updates to genmon.py to make they behavior a little more predictable considering the unknowns listed above so you should not get the odd notifications of transfer switch status.

You should still get email notices when your engine or switch state change, just not your transfer switch until we get these register issues ironed out.

I have a couple of ideas but I need to think on this for a day to determine the best approach and the order that we do things. After I get the pictures I will likely have a better idea of what to do first/next. At some point we will likely need to do a register capture with the generator running.

jgyates commented 7 years ago

OK. I got your photos. That helps a lot. The photos pointed out that your generator uses the Nexus controller, not the Evolution Controller. The good news is that that let's us know what to expect regarding functionality from your system. The Nexus controller is an older controller in terms of the Generac controller lineage and as a result it does not support as many features according to their documents. Where this leaves us is that out of the list of items that were not working on your generator with genmon.py, it looks like many of the problems are just not supported by the Nexus controller, for example there is not maintenance log on the nexus and I do not think it uses error codes in the display. We still have the following two issues:

1) Can we get setting the exercise time to work (including quite mode)? 2) Can we find the location of the Input and Relay Output Registers in the Nexus controller. These two registers are reflected in the menu (Debug-> Test -> Outputs and Debug -> Test -> Inputs). The values of these bits in these two registers are defined in the Air Cooled Diagnostic Manual 3) The format of the Alarm Log does not appear to be correct.

Air Cooled Diagnostic Manual Page 42.

This like is also in the project wiki.

This manual is for both the Nexus and Evolution Controllers but page 42 appears to just reference the Evolution controller, however the bit positions in the pictures you sent some what line up with the descriptions in the document (at least the digital outputs do, i.e. if you battery is over 13V it is probably charging and the corresponding bit, should be active in the output register). The register number for the Evolution controller is apparently not the same as the Nexus for the input and output registers. When I look for the bit pattern of "0100" in a full dump of the registers I do not see this value so I am a bit unsure if the Nexus supports this feature in the register map.

I added a new option to genmon.conf that allows selecting Nexus vs Evolution Controller and made the functionality in genmon.py match what is expected from each type of controller. I am still thinking about ways to solve our outstanding issues. I will let you know how is is going.

I updated your system with the latest genmon.py and genmon.conf (I kept your site name the same)

JeremyStevenson commented 7 years ago

Thanks! I didn't see any emails come through - did you turn email notifications off?

I'm wondering if we can't get the manual start or changing the exercise time to work, perhaps we could tie the "transfer" a.k.a. "transfer sig" line (#23) to a GPIO on the Pi. If I'm not mistaken, that's the line that the transfer switch uses to start the generator.

Let me know if you need me to start/stop the generator and collect register information.

jgyates commented 7 years ago

I edited the mymail.conf file to my email address when I was testing, then I changed it back to your email address once I was done testing.

Regarding the GPIO to transfer line #23, possibly, but it may conflict with the transfer relay controlled by the controller. Also, I believe the GPIO is open collector, I have not looked at the schematic of the transfer pin so it may need a transistor to drive it properly. Setting GPIO is pretty easy on the PI, I use it to control my gate but I use a daughter card that has relays on it to properly drive the gate controller. Daughter card is PiFace2:

https://www.element14.com/community/docs/DOC-69001/l/piface-digital-2-for-raspberry-pi

Tonight or tomorrow I will try to setup a test on the your system. Once it is setup I will let you know, then you can, at your convience turn on the generator and genmon.py will notify us of changed registers. These are features that are documented in the code but not in the readme as they mostly helped me reverse engineer the bits in the registers.

The experimental remote start code in genmon.py is in a function called SetGeneratorRemoteStartStop(). It is not called by another code at the moment. There are two types of registers in the controller 1) normal registers and 2) indexed registers. The indexed registers are occupied by registers 0003 and 0004. You write the value to 004 and the register number to 0003. Genmon.py uses the index registers to set the exercise time (using the legacy method I added recently). The indexed method does not appear to work on your Nexus controller to set the exercise time so I am not sure if it will work using indexed registers to remotely start. We can try though. Index register 0006 controls exercise time.date Index register 0003 controls remote start (data written 0001 to start,I believe ). I accidentally wrote a 0x0203 top this register and my generator started. I turned it off manually. Index register 0002 controls remote transfer switch (Not sure of the data here )

I can write a routine that will allow us to test this via ClientInterface.py.

jgyates commented 7 years ago

I have setup a test with genmon.py running in the background. Here is how the test works:

The registers are defined in a few list (e.g. self.BaseRegisters[]). These list have elements defined like this: "0000" : [2, 1], where "0000" is the register number, "2" is the length of the register, and "1" is a flag to monitor the register for changes. I added a bunch of registers to the list (up to 0x60, plus a few select others and made most of them monitor. If any of the monitored registers change I will get an email with some base status and a full register dump. I loaded the program as a background process "sudo python genmon.py &". To kill the process type "sudo pkill -f genmon.py". I also have the emails set to come to me but let me know if you want a copy or to have them sent to you first. I did not check in this version as it is a work in progress / experimental version with a lot of registers added to the list so it does slow performance / responsiveness down. Also, I did have a few registers that appear to be floating inputs (i.e. they change randomly and all of the floating values change to the same value) so I removed them from the test. I saw this on my system when doing the initial development.

When ever you can at a minimum you can turn the generator on, let it run for about a min, then power it off. A better test would be to disable power at your meter box, let the generator detect the outage, start the generator and throw the transfer switch without any other intervention. Let the gen power up and run the house in backup power for about one min then flip the switch at the meter back to utility. If you are unable to run the optimal test that is OK. Just giving you all the options.

JeremyStevenson commented 7 years ago

Will do the full test tonight. Thanks!

On Mar 30, 2017 10:11 PM, "jgyates" notifications@github.com wrote:

I have setup a test with genmon.py running in the background. Here is how the test works:

The registers are defined in a few list (e.g. self.BaseRegisters[]). These list have elements defined like this: "0000" : [2, 1], where "0000" is the register number, "2" is the length of the register, and "1" is a flag to monitor the register for changes. I added a bunch of registers to the list (up to 0x60, plus a few select others and made most of them monitor. If any of the monitored registers change I will get an email with some base status and a full register dump. I loaded the program as a background process "sudo python genmon.py &". To kill the process type "sudo pkill -f genmon.py". I also have the emails set to come to me but let me know if you want a copy or to have them sent to you first. I did not check in this version as it is a work in progress / experimental version with a lot of registers added to the list so it does slow performance / responsiveness down. Also, I did have a few registers that appear to be floating inputs (i.e. they change randomly and all of the floating values change to the same value) so I removed them from the test. I saw this on my system when doing the initial development.

When ever you can at a minimum you can turn the generator on, let it run for about a min, then power it off. A better test would be to disable power at your meter box, let the generator detect the outage, start the generator and throw the transfer switch without any other intervention. Let the gen power up and run the house in backup power for about one min then flip the switch at the meter back to utility. If you are unable to run the optimal test that is OK. Just giving you all the options.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jgyates/genmon/issues/4#issuecomment-290596198, or mute the thread https://github.com/notifications/unsubscribe-auth/AJaebO1ycg3gWSH2M4q1aSXuHQ3oxUtyks5rrGDKgaJpZM4Mpr5w .

JeremyStevenson commented 7 years ago

Just cut utility power. Will leave of for 1 min.

On Mar 31, 2017 5:28 AM, "Jeremy Stevenson" jeremy@formulatrix.com wrote:

Will do the full test tonight. Thanks!

On Mar 30, 2017 10:11 PM, "jgyates" notifications@github.com wrote:

I have setup a test with genmon.py running in the background. Here is how the test works:

The registers are defined in a few list (e.g. self.BaseRegisters[]). These list have elements defined like this: "0000" : [2, 1], where "0000" is the register number, "2" is the length of the register, and "1" is a flag to monitor the register for changes. I added a bunch of registers to the list (up to 0x60, plus a few select others and made most of them monitor. If any of the monitored registers change I will get an email with some base status and a full register dump. I loaded the program as a background process "sudo python genmon.py &". To kill the process type "sudo pkill -f genmon.py". I also have the emails set to come to me but let me know if you want a copy or to have them sent to you first. I did not check in this version as it is a work in progress / experimental version with a lot of registers added to the list so it does slow performance / responsiveness down. Also, I did have a few registers that appear to be floating inputs (i.e. they change randomly and all of the floating values change to the same value) so I removed them from the test. I saw this on my system when doing the initial development.

When ever you can at a minimum you can turn the generator on, let it run for about a min, then power it off. A better test would be to disable power at your meter box, let the generator detect the outage, start the generator and throw the transfer switch without any other intervention. Let the gen power up and run the house in backup power for about one min then flip the switch at the meter back to utility. If you are unable to run the optimal test that is OK. Just giving you all the options.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jgyates/genmon/issues/4#issuecomment-290596198, or mute the thread https://github.com/notifications/unsubscribe-auth/AJaebO1ycg3gWSH2M4q1aSXuHQ3oxUtyks5rrGDKgaJpZM4Mpr5w .

JeremyStevenson commented 7 years ago

Power restored.

On Mar 31, 2017 6:59 AM, "Jeremy Stevenson" jeremy@formulatrix.com wrote:

Just cut utility power. Will leave of for 1 min.

On Mar 31, 2017 5:28 AM, "Jeremy Stevenson" jeremy@formulatrix.com wrote:

Will do the full test tonight. Thanks!

On Mar 30, 2017 10:11 PM, "jgyates" notifications@github.com wrote:

I have setup a test with genmon.py running in the background. Here is how the test works:

The registers are defined in a few list (e.g. self.BaseRegisters[]). These list have elements defined like this: "0000" : [2, 1], where "0000" is the register number, "2" is the length of the register, and "1" is a flag to monitor the register for changes. I added a bunch of registers to the list (up to 0x60, plus a few select others and made most of them monitor. If any of the monitored registers change I will get an email with some base status and a full register dump. I loaded the program as a background process "sudo python genmon.py &". To kill the process type "sudo pkill -f genmon.py". I also have the emails set to come to me but let me know if you want a copy or to have them sent to you first. I did not check in this version as it is a work in progress / experimental version with a lot of registers added to the list so it does slow performance / responsiveness down. Also, I did have a few registers that appear to be floating inputs (i.e. they change randomly and all of the floating values change to the same value) so I removed them from the test. I saw this on my system when doing the initial development.

When ever you can at a minimum you can turn the generator on, let it run for about a min, then power it off. A better test would be to disable power at your meter box, let the generator detect the outage, start the generator and throw the transfer switch without any other intervention. Let the gen power up and run the house in backup power for about one min then flip the switch at the meter back to utility. If you are unable to run the optimal test that is OK. Just giving you all the options.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jgyates/genmon/issues/4#issuecomment-290596198, or mute the thread https://github.com/notifications/unsubscribe-auth/AJaebO1ycg3gWSH2M4q1aSXuHQ3oxUtyks5rrGDKgaJpZM4Mpr5w .

jgyates commented 7 years ago

I received about 260 emails for the test. You should have copies in your sent folder. I found a couple of things that look promising regarding the transfer switch and output register. I also found one issues with the Frequency registers that is easily fixed. Apparently the Nexus controller only reports frequency in whole numbers (Evolution uses the tenths place). I did get notified as expected for starting, running, stopping and power loss / restore. There is also one bit in register 0001 that I have never seen set before. This is the main status register so we will likely want to figure that one out. It may require you taking a quick video of the display while you do an manual (unless you know a way to take a video of the generator while you flip the power off). genmon.py sends and email when ever the contents of register 0001 changes. If we don't know what the bit is we indicate in the email as UNKNOWN however the controller display normally outputs what state the generator is in so this is the reason for the video. I will reset the test today (really just refine it a bit to weed out registers we know we don't need) and let you know when the next test is ready then you can do it when you have the time.

I will send more info as i go though the emails.

Thanks Jason

jgyates commented 7 years ago

I made some more tweaks for the Nexus controller. setexercise now works. setquiet does not but I know why, it must also use the legacy write method. This is good news that the indexed writes are working. I made the startup code in genmon.py force uselegacysetexercise to True if evolutioncontroller is False in genmon.conf. This will make sure that if someone selects a Nexus controller it will automatically user the legacy write for setexercise. It still has the issue of being off by on min from time to time but it does appear to work. I also added a line in the "monitor" command that displays the controller type and liquid vs air cooled setting currently in use. I also added descriptions to genmon.conf and mymail.conf in the comments of each file.

I am not having a lot of luck finding the output relay registers. The register test we did was helpful but did not solve all of our issues. I added some code to cleanup the remote start stuff. We will likely want to test that with you on site since the code to turn it on may work but the code to turn it off might now.

I may try to poke around on getting the alarm log working. The start / stop log works as expected per your photos.

JeremyStevenson commented 7 years ago

Ok. I'll be on site in about 3 hours. Can you give me some specific tests to run?

Are you going to add a ClientInterface command to start/stop? Or, is it already there just not documented?

jgyates commented 7 years ago

It is running now. I have it setup to monitor register 0001 so we can do the following tests:

1) manual exercise, take video or look at the controller display. What we are looking for here is the sequence of notices the display shows relative to the status of register 0001 as the generator cranks -> Starts -> Runs -> Cools Down-> Stops. My guess is that it will show this sequence. I think that the bit is crank or possibly starting with warning (you have a check battery message in your log) .We have decided Start, Run, exercise, etc, but there is one bit that I don't have a definition for when this bit is the only one set.

2) Do a power cycle on the utility line like you did before.

3) I have added the command line test for remote start (not documented, except here): generator: setremote=RemoteStart generator: setremote=RemoteStop generator: setremote=TransferUtility generator: setremote=TransferGenerator

These are very experimental so there is a likely hood that you may have to stop the generator manually but it will at least give us a starting point, start and stop may be reversed, also as you issue these commands look at the display and make notes on what the controller shows since we may be matching register values to these display strings.

jgyates commented 7 years ago

I got the emails for your weekly exercise. No need to do test #1. The unknown bit value of register 0001 does not appear to show up on exercise, only power fail apparently.

JeremyStevenson commented 7 years ago

Snow storm and family duties. May not get to this until tomorrow, sorry.

On Apr 1, 2017 2:57 PM, "jgyates" notifications@github.com wrote:

I got the emails for your weekly exercise. No need to do test #1 https://github.com/jgyates/genmon/issues/1. The unknown bit value of register 0001 does not appear to show up on exercise, only power fail apparently.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jgyates/genmon/issues/4#issuecomment-290940015, or mute the thread https://github.com/notifications/unsubscribe-auth/AJaebKUsXmkfMG7DcxPjsqDzETLPjsk5ks5rrp4UgaJpZM4Mpr5w .

jgyates commented 7 years ago

No problem. We are on your schedule since is is your generator :)

JeremyStevenson commented 7 years ago

At 8:57 am Eastern time I ran all the commands in #3. No effect.

No dice on setexercise also.

About to kill utility power (9:07 eastern time)...

jgyates commented 7 years ago

I just logged in to you system and used set exercise. The maint commands shows it updated. This time, instead of changing the exercise time back to Saturday at 8:00 I left it to Sunday at 12:30, it should send emails. How are you determining setexercise is not working for you? I think we may have a difference in how we are validating things. It may take one or two seconds for the setexercise command to show up. If you are still on site I can log in again and try the commands to remote run.

Here the output I got:

-generator: maint Maintenance:

Exercise: Exercise Time: Saturday 08:00 Quite Mode On

Service: Service Scheduled: Next service in 199 hours Total Run Hours: 50 Firmware : V1.43, Hardware : V1.02

-generator: setexercise=Sunday,12:30 Set Exercise Time Command sent (using legacy write) -generator: maint Maintenance:

Exercise: Exercise Time: Sunday 12:30 Quite Mode On

Service: Service Scheduled: Next service in 199 hours Total Run Hours: 50 Firmware : V1.43, Hardware : V1.02

JeremyStevenson commented 7 years ago

Ok, I tried again. I've been trying to use the setmaint command to kick off the generator. So, I wanted to set it for 1 or 2 minutes from the current time. That doesn't work!

It seems that the startup time needs to be at least 2 or 3 minutes from the current time.

... and my generator just started, so indeed setexercise works but it needs to be 2 or 3 minutes from the current time.

jgyates commented 7 years ago

That makes sense. The legacy method of setting the exercise time requires you to write values to the register that are obtained by taking the time you want to exercise and subtracting it from from the current time so there may be some logic in the firmware that prevents setting a time less than a few minutes from now.

JeremyStevenson commented 7 years ago

So, looks like most things are working except for the experimental RemoteStart and RemoteStop. Great!

I'm starting to believe that remote start is NOT possible for the Nexus controller. Would you agree with this?

jgyates commented 7 years ago

I think there is a way to do this, I just don't have the right bit combinations yet. I will do some testing and put another version out there. The reason I think that it will work is that Generac sells remote wireless remotes that appear to work on both Evolution and Nexus units. Like this one: https://www.electricgeneratorsdirect.com/Generac-6664/p13356.html

and this one: https://www.powerequipmentdirect.com/manuals/nexusadvwirelessmonitor_spec.pdf

jgyates commented 7 years ago

It will probably be later this evening or tomorrow before I can get to do additional testing. I will let you know when I have something to test on your end.

jgyates commented 7 years ago

OK. I think I found an issues that would keep it from working before. I also did some additional testing on my generator and I think I have the remote stuff working. I only have it working via ClientInterface, not the email interface at the moment. The commands are

to start the generator running full speed and output (i.e. not quiet mode) -generator: setremote=start

to start the generator at full speed and output and then switch the transfer switch so the house will run from the generator -generator: setremote=starttransfer

to start the generator running low speed / low output (i.e. quietmode exercise) -generator: setremote=startexercise

to stop any of the above commands

Note: if you run the starttransfer the generator will run in cool down mode for a min or so after you issue a stop command. This is built into the firmware.

I have tested all four of these commands on my generator and it appears to work. Let me know how it works on yours.

I also made the controller and legacywrite selection in genmon.conf obsolete by auto-detecting the controller based on the register contents. If you uncomment these in your conf file you can override the auto-detect.

I updated your copy of genmon.conf and genmon.py and it is loaded and running. You should just have to exercise the commands.

JeremyStevenson commented 7 years ago

Oh! Woo hoo! All those commands worked. Great job!

Indeed, it took a couple of minutes to stop after the transfer.

It looks like if you settransfer to earlier the same day then it won't take. E.g. I can't set to Sunday at 8:00 am right now. It works for setting to Saturday at 8:00.

This is a way better solution now than anything offered by Generac!

Let me know when you want to transfer email events back to me and change the password back. Thanks again.

Next, I'm wondering about a cell phone data backup...

jgyates commented 7 years ago

Thanks for the kind words. I am glad it is working for you and I really appreciate you letting me test on your system. You can change mymail.conf back to your email address and change the ssh password.

I updated the version of genmon.py in the github repository. The version you are running only supports setremote commands on the command line, the version online supports both email and command line.

I will close this issue now that we have things working. Feel free to open another one if you see any issues or email me if you have any thing that does not work well for issues tracking.

I will add a note about setting the exercise time to the docs. Thanks for the feedback.

My 2 cents on phone backup: I used to used Titanium backup a while back in early android days, but it requires root on your phone. I don't run root any more (no real need anymore in my view). I also used Helium backup, it does not require root. both are in the play store (assuming you use android). I don't run a backup on my phone anymore since goolge backups up most things now.

Thanks again for you help. Jason

jgyates commented 7 years ago

Closed, issues resolved.