search

jgyates / genmon

Generac (and other models) Generator Monitoring using a Raspberry Pi and WiFi
GNU General Public License v2.0
385 stars 79 forks source link

Industrial Gens #98

Closed darkstar2002 closed 6 years ago

darkstar2002 commented 6 years ago

Has anybody tired using this on the Industrial versions? We have a 550kw, 400kw, and a 175kw.

I know they have a db9 serial port on the front of the controllers.

jgyates commented 6 years ago

Not that I am aware of. The software is designed to work with Either Nexus Controller or the Evolution Controller. After a quick seach it looks like the industrial version use the H-Panel for the controller. This is not supported by the software, however the H-Panel does appear to have a documented modbus interface which should make it easier to support.

jgyates commented 6 years ago

@darkstar2002

If you are interested in industrial version being supported, I have added a program to the repository that would provide the first step in adding the industrial generators as a supported generator. It would require you to create a cable and run a program connected to the generator to get the contents of the controller registers for each generator. From there I would start the process of adding the code to support this generator. I would anticipate this process could take a few months, depending on the difficulty involved in interpreting the existing documentation online.

Let me know if you are interested and I can provide additional instructions.

darkstar2002 commented 6 years ago

I am interested in doing this. It may be a week or so before I will have the time to get to the generators.

jgyates commented 6 years ago

@darkstar2002 Here are the first steps:

1) Download this manual: https://gensetservices.com/wp-content/themes/gensetservices/PartsPDFs/0F3750.pdf 2) Download the genmon repository onto the raspberry pi or PC 3) In the repository, move the file /genmon/OtherApps/modbusdump.py up one directory cd ./genmon/OtherApps cp ./modbusdump.py ..

We will run this program with the cable attached to the controller to validate communication and get the contents of the modbus registers. Once this is validated and we have the contents of the registers I can begin integrating support for this controller in the software.

4) Attach a cable from the 9 pin port on the the front to a computer that can run python (raspberry pi, or PC). Attach the other end to the raspberry pi or PC

From here we will run the modbusdump program :

The syntax for the program is:

python mobusdump.py -r -p -a -s -e

Note the -a parameter is in hex and the -s and -e params are in decimal.

If you refer to the manual in the link above you can get the modbus slave address (page 21, default could be 100 or 64 hex), baud rate, see page 21). The serial port is dependent on if you are on a PC (windows or linux) or raspberry pi (/dev/serial0).

Once we get all of this info we will need to select a start and end register to probe (parameter -s and -e). For our initial test lets start with -s 140 and -e 142:

 python mobusdump.py -r 9600 -p /dev/serial0 -a 64 -s 100  -e 101

Copy and past the output of these test to this thread. Let me know if you have any questions.

darkstar2002 commented 6 years ago

I have tried running this on 2 pc's one windows and one linux. I get an error about the libs not found. I did move the file up one directory.

jgyates commented 6 years ago

what is the error? Is the file modbusdump.py in the same directory as genmon.py? Please make sure you have the latest version of the of the code as well.

jgyates commented 6 years ago

I forgot to mention that you will need to install the python serial library. Run these commands from the linux to install all the libraries needed for genmon (for this test, you only need crcmod and python-serial, and pip to install the libraries):

 sudo apt-get update
 sudo apt-get install python-pip
 sudo pip install crcmod
 sudo pip install configparser
 sudo apt-get install python-serial
 sudo pip install Flask
liltux commented 6 years ago

I am a big fan of this project and thought I could offer some help in this area. It is the modbus mapping for generac industrial generators. see the zipped file.

Modbus Map Generac Industrial.zip

MichaelB2018 commented 6 years ago

I would love to help as well to improve this further, but I have a residential unit. @liltux, do you have that maps?

darkstar2002 commented 6 years ago

Got things working. When trying different start and stop ranges I get a error reading device: byte must be in range(0, 256)

jgyates commented 6 years ago

Great! Couple of questions:

1) What is the command line your are using 2) The program will log errors to a file named mymodbus.log in the current working directory. Can you include this file? 3) Does your controller screen tell you the modbus address? If yes what is the value?

darkstar2002 commented 6 years ago

python.exe modbusdump.py -r 9600 -p COM4 -a 100 -s 140 -e 142 The log does not add anything when this error comes up. If I leave my adapter off it says can not open the port

Slave address is 100 9600 buad

jgyates commented 6 years ago

I just checked in an updated version of mymodbus.py, the library that the program is using to handle the modbus details. This should give me more info on the problem. This error is caused by trying to put a value larger than 255 in a byte array. I am not sure if this is happening on the send or receive but the new version of mymodbus.py will let me know so I can correct this problem. Update to the latest software and let me know the results. It should put the error with more details on mymodbus.log. Let me know if you have any questions.

jgyates commented 6 years ago

Another question: What version of python are you running? You should be able to type:

 python -V

and it will show you the version. There has been little testing with version 3.x. 2.7 is recommended.

darkstar2002 commented 6 years ago

I am running 2.7 on windows. I am working on setting up a virtual linux for the laptop.
the program says ok no info returned. Here is log file. mymodbus.log

jgyates commented 6 years ago

ok. I think I see the problem. Give me a few min to fix it.

jgyates commented 6 years ago

OK. Here is the the issue: My command like is a little confusing. Sorry about that. The modbus address should be in hex so if you address is 100 decimal we should put 64 (hex) on the command line. Use this command line:

 python.exe modbusdump.py -r 9600 -p COM4 -a 64 -s 140 -e 142
darkstar2002 commented 6 years ago

Ok got it working.

Address is : 64 Start Register : 140 Start Register : 142 008c:0000 008d:007d OK

jgyates commented 6 years ago

Excellent! Now that we have validated communications lets run this command line:

 python.exe modbusdump.py -r 9600 -p COM4 -a 64 -s 0 -e 400

You can redirect this to a file if you like and just send the file:

 python.exe modbusdump.py -r 9600 -p COM4 -a 64 -s 0 -e 400 > out.txt

This will give me what I need to emulate your generator. Send me this once you can run it.

When you can, I will also need a dump of the registers with the generator running.

Ideally it would be good get the test one each of your generators (only need the not running test on the others)

I can get something up and running with this data. I may need additional data at some point but I will let you know.

darkstar2002 commented 6 years ago

Here you go. 400 off and 2 400 running. They are in manual mode to make them run. 550 off is from our 550kw. It might be a few days before I can get to our 175kw. 550_off.txt 400_off.txt 400_on.txt 400_on2.txt

jgyates commented 6 years ago

Wonderful. Thanks! I will let you know how things progress.

darkstar2002 commented 6 years ago

Here is our 175kw natual gas model. 175_off.txt

jgyates commented 6 years ago

@darkstar2002

So far things are going well. Below is what I am getting from the register dump so far (plus alarm conditions. This is from the running 400kw version that you sent:

Status : 

Engine : 
    Switch State : Manual
    Engine State : Running. Ready for Load. 
    Output Power : 0.00 kW
    Output Power Factor : 1.00
    RPM : 1805
    Frequency : 60.10 Hz
    Throttle Position : 1800 Stp
    Coolant Temp : 118 F
    Coolant Level : 794 Stp
    Oil Pressure : 56 psi
    Oil Temp : 0 F
    Fuel Level : 48
    Oxygen Sensor : 0
    Current Phase A : 0.00 A
    Current Phase B : 0.00 A
    Current Phase C : 0.00 A
    Average Current : 0.00 A
    Voltage A-B : 21.20 V
    Voltage B-C : 21.30 V
    Voltage C-A : 21.30 V
    Average Voltage : 21.30 V
    Engine Hours : 418.1 H
    A-F Duty Cycle : 7.30

Battery : 
    Battery Volts : 27.82 V
    Battery Charger Current : 0.00 A

Line State : 
    Transfer Switch State : Unknown

Time : 
    Monitor Time : Wednesday May 2, 2018 22:39:00
    Generator Time : Tuesday May 1, 2018 11:45:02

If things like run hours and coolant temp do not match, let me know. I made my best guess at info that has decimal places. In general let me know if values are out of range.

There are still a number of items I have yet to locate( utility voltage, log registers, service info). What would help would be a dump of a larger range of registers. When you have time can you run the following:

python.exe modbusdump.py -r 9600 -p COM4 -a 64 -s 0 -e 17000 > out.txt

Also, do a dump and make note of the exercise time, then change the exercise time and do another dump. That will help narrow the search a bit.

jgyates commented 6 years ago

Found a mistake, Engine hours should be : Engine Hours : 105.5 H

darkstar2002 commented 6 years ago

I do know that the voltage should be 200 something. I will try to get to the generator today if the weather will corporate.

jgyates commented 6 years ago

I changed the voltage output to not divide by 10 so it should read 212 in stead of 21.2

darkstar2002 commented 6 years ago

We lost power for a little bit. Here is our 550 running on lost utility power. I am also attaching the e version which is what I could get from your 17000 request. Had to cut it off after 20min due to it starting to rain again. 550_on.txt 550_on_e.txt

darkstar2002 commented 6 years ago

Also the engine hours on the 550 is 97.7. current kw load on this run was around 160-210kw.

jgyates commented 6 years ago

@darkstar2002

I have an initial alpha type version up and running under simulation using your supplied dump files. If you want to test you can do the following:

Checkout the latest and install the software as you would normally per the wiki.

Once complete with the installation edit /etc/genmon.conf add/edit the following lines change

address=9d

to

address=64

add this line to the end of genmon.conf

controllertype=h_100

Things that are missing or not working yet:

darkstar2002 commented 6 years ago

I was able to look at some of the stuff. Battery and Fuel level works. I could not see what it was doing when the generator is on. I will be ordering hardware soon to hook these up permanently so I do not have to go out with a laptop to try things out.

jgyates commented 6 years ago

Sounds good. I am working on some things now that should make getting this working easier, for example if you enable email on the system I have an option built in that if you enable it, it will email me the large register dumps automatically so you will not have to run a specific test for that. Of course if you don't want email enabled we can do it the manual way. Also, you can upgrade each system to the latest software via the web interface (assuming the pi has internet access) so that will make updating as I add new functionality.

darkstar2002 commented 6 years ago

Getting one of the pi's ready to on the big generators. I get a LoadConfig() error from genserv.py

jgyates commented 6 years ago

Have you copied the genmon.conf file to the /etc directory? This is the where the main configuration is stored for both genmon.py and genserv.py

Once you have /etc/genmon.conf in place. Add the following line to your /etc/genmon.conf file:

 controllertype=h_100

Also, modify the value in /etc/genon.conf for the modbus address, change if from 9d to 64

 address=64

This will enable the support for your industrial generators. Once it is up and running you should be able to access the web interface, enable email, etc. for each generator you have it connected to. Once all of your settign are configured and email is working you can uncomment and modify this value to /etc/genmon.conf

 enabledebug = True

This will enable code that will read the registers in bulk. If you have "Automated Feedback" enabled in the web interface settings an enabledebug=True in your conf file the software will read the registers and email them to me. This should help with the reverse engineering of the registers on this generator.

I know that based on the registers that you sent the generators appear to be configured slightly different in terms of the transfer switch. The bits that report the transfer switch status are different on some of them so this is something we will need to look at. From the docs it looks like it depends on the type of transfer switch that is used. I mentioned other items that are know to be an issue in this thread above.

Let me know if you have any questions or problems.

darkstar2002 commented 6 years ago

For what ever reason its working now. Is there a way to force a email out of your system? I need to test that it is working. I have a complex firewall / email system here and just want to make sure its working.

darkstar2002 commented 6 years ago

Never mind its all working. I have the debug turned on. Right now I only have our 400 hooked up.

darkstar2002 commented 6 years ago

I had to shutdown the enable debug. It was throwing serial errors and would not update the dashboard

jgyates commented 6 years ago

The ClientInterface.py program (you can run it from a PC or the pi) will let you enter a command that will send your log files to me.

If run on the pi type:

 python ClientInterface.py

If run on a PC include your IP address of the PI you want to communicate with:

 python ClientInterface.py IPADDRESOFPI

Once ClientInterface.py us running (type "exit" to exit the program) you can type:

 generator: sendlogfiles

This will email your error logs to me and allow me to get a better insight into any problems with ther debug thread.

jgyates commented 6 years ago

I receive your log files. For some reason the serial device is reporting this error: "device reports readiness to read but returned no data (device disconnected or multiple access on port?)" This error appears to be causing packets to be dropped since the software will attempt to reset the port if it detects an error.

It is possible that another process is accessing the USB serial port? Is this a relatively fresh install of the operating system?

jgyates commented 6 years ago

From the console type this command:

 sudo fuser /dev/ttyUSB0

This should return a single number (the process ID of the program using the port). If multiple numbers are returned then we have an problem. If one is returned then I need to keep looking...

jgyates commented 6 years ago

More question:

What is the model / manufacturer of the USB serial device? If the power requirements for the USB device are too large it could cause problems.

Which Pi version are you using?

Also make sure the usb device is making a good connection and there is not excessive strain on the connector.

darkstar2002 commented 6 years ago

I only get the errors when the debug is true. its a plugable serial adapter. Pi 3 fuser only shows one pid for debug on and off.

I had debug off for several hours and had not had any errors come up.

jgyates commented 6 years ago

OK. What is odd is that I can run the debug thread with no errors. I have a lock in the code that prevents the debug thread from stepping on the normal thread that reads the modbus registers and this is common code for the H-100 and the residential generators so I am fairly confident the debut thread alone is not the issue. I could be the additional throughput the debug thread is putting on the port is causing problem with the USB serial device or the power level is maxing out on the pi.

A couple of things we can to to narrow this down:

1) Run the command: vcgencmd get_throttled

If we get non zeros back we could have a power problem.

2) Open up a couple of consoles so we can monitor some of the system logs. Each one of the commands you can stop with Ctrl+C and each command should be run in a separate console session.

If there is a sudden disconnect of the USB device it should log it in one of these logs.

I looked up the power requirements of hte pluggable USB device, it is 100mA. The limit is 500mA for a standard PI 3 I think so unless you have other devices connected the power on the USB should be fine.

How are you powering your pi? I assume you have a ground connection on your serial cable?

I will try to run some tests on my end to see if I can duplicate this issue.

darkstar2002 commented 6 years ago

Running the pi off poe. serial usb connects to a regular serial cable that goes to the gen controller. I have no usb disconnects running debug. the throttled command is 0x50000 I only get a undervolt issue right at startup but it corrects almost immediately. Its completely good by the time the login screen is up.

jgyates commented 6 years ago

out of curiosity, on the Monitor page, what is the average transaction time?

darkstar2002 commented 6 years ago

Average Transaction Time : 0.0276 sec

With debug off.

jgyates commented 6 years ago

Interesting, this is a faster packet response than the residential version (.0309 is typical for the residential) . I guess they packed a speedier processor in the big gens..

What is the Packets per second?

jgyates commented 6 years ago

I checked in an update to the software. I don't think this will change the behavior that we are seeing, but it was a needed updated to the logic that reads the registers. You can update the software from the "Monitor" page.

when you get a chance, run the debug test again while monitoring the syslog, messages and dmesg for errors.

jgyates commented 6 years ago

I made another change to the code that may help. I slowed down the debug thread, it now sleeps 0.25 seconds between modbus reqeust. I should take a bit longer to read the full set of registers but if it works it should be able to do it's thing unattended. Let me know how it goes. on my system (with an average transaction time of 0.309 seconds) it took about 30 min. The way the debug thread works is that it reads registers 0 - 0x2000 (8192 decimal) for 5 passes, each pass takes about 30 min with the current build. After each pass the registers are compared to see what has changed.

Another question. Can you read the utility line voltage from your generator display? With the base registers I have, I have not been able to find registers that correspond to the utility line voltage. Depending on how the transfer switches are setup it is possible that some configurations will not have the utility voltage available to the generator controller.

jgyates commented 6 years ago

I did some digging on the error that is reported on the serial interface "device reports readiness to read but returned no data (device disconnected or multiple access on port?)" . According to the comment in the code for the serial python library that is generating the error, this error occurs when the device is disconnected. My guess is that it is a power issue, possibly an interaction between you POE injector an the serial USB device's power consumption. The change I checked in may work around this issue, but you could still have problems later. You may consider trying to migrate away from the USB device and move to using the pi's internal serial port as this should greatly reduce the likelihood of this error. Here is the link to the python serial code that generates the error:

https://github.com/pyserial/pyserial/blob/master/serial/serialposix.py#L520

jgyates commented 6 years ago

I added email notifications in the H-100 code so you should see and email when it starts, when the engine state changes and when an alarm occurs. Emails during an outage will work once I can determine the utility line voltage.