search

genielabs / HomeGenie

HomeGenie, the programmable automation intelligence
https://homegenie.it
GNU General Public License v3.0
388 stars 154 forks source link

Homegenie service stops responding #361

Closed ngineerken closed 5 years ago

ngineerken commented 5 years ago

Expected behavior

HomeGenie just runs

Actual behavior

HomeGenie runs about 3 days before becoming unresponsive

Steps to reproduce the problem starting from a fresh install

  1. Start the Pi
  2. Services are running
  3. After 3 days HG has stopped responding

Specifications

Additional configuration notes

the Pi is also running as a weather station using weewx, this is why i use a PiDrive as i was burning through SD Cards

tuicemen commented 5 years ago

HG is on 1.2-Stable.35 You may wish to update to it and test if you still have that issue. I also had that issue using a 3B+ and Version: 1.1-stable.527 I resolved it by creating a cron event to reboot every night. I never did check if it was HG (sudo systemctl status homegenie.service) or my Wi-Fi that just wasn't reconnecting. Pis seem to not reconnect if connection is lost without some extra code added.

ngineerken commented 5 years ago

I'm at Version: 1.2-stable.35 Setting a reboot cron job is not the answer to an issue that has arisen , it's only a band-aid to the bug.

mralapete commented 5 years ago

@ngineerken agreed. Band-aid fixes of no value. Have you got logging enabled. If you have disable it and see what the results are. Also what version of Mono are you using. I've had similar results in the past and changing SD card appeared to resolve the issue the issue temporarily. I'm now booting from a USB stick and things seem to be more stable.

tuicemen commented 5 years ago

@ngineerken your original post stated you were at Version: 1.1-stable.527 and my work around was what I did for that version. I've not had that issue since moving to genes newer versions and moving to a newer Mono version. It's only a bug if others are seeing it, Gene put out version 1.2-stable 36 today so you might try that if you've not already.

tuicemen commented 5 years ago

@mralapete, Band aid fixes (work arounds) are of value if they get ones system working until a permanent fix is found. @ngineerken besides the log file what does a "sudo systemctl status homegenie.service" report when this happens?

mralapete commented 5 years ago

@ngineerken if after turning off logging shows no improvement to yiur system I suggest that you run your HG server in debug mode and at the point of crash post your logs here so we can see exactly what is causing your crash.

@tuicemen, you're missing the point. @ngineerken is running a stable version v35 of HG and it is crashing after a number of days. Upgrading to v36 will not solve his problem. We need to see his server logs to determine his issue. It could be one of a number of issues here.

tuicemen commented 5 years ago

@mralapete no I'm not missing the point. Originally @ngineerken posted they were running 1.1-stable.527 maybe that was a typo which they edited to fix after my first response.

Since Gene never posted what was fixed or added in v36 how can you be sure updating will not fix the issue. @ngineerken never posted HG was crashing just that the service was not responding. A log file may or may not reveal the issue as may a status request to the service.

ngineerken commented 5 years ago

I had logging on, now disabled. I also update to v36 and now lets see how things go.

Thanks for all the help

tuicemen commented 5 years ago

Did the logging show anything (errors)?

mralapete commented 5 years ago

@tuicemen at the risk of repeating myself the logs will be helpful to determine what exactly is going on here. Unresponsive would indicate a crashed program. If you feel you can troubleshoot a system without them, particularly a server I will take my had off to you. Personally I prefer the log route rather than the crystal ball

I imagine Gene and others would agree with this conclusion too and remember upgrading may only be hiding a bigger issue if it's not investigated properly. The fact that others have not reported an issue similar to @ngineerken leads me to think his issue is particular to his system only. Guesswork will not solve that problem.

I

tuicemen commented 5 years ago

I'm not sure why you seem to always think I don't recommend logging! There is no use trying to debug a old version which I assumed @ngineerken was running from his original Post. I use to hate users posting bug reports for old versions of my software. Especially when I knew the issue wasn't present in the new version. That being said you'll notice I asked if @ngineerken log showed anything!

ngineerken commented 5 years ago

Well turning logging off did not help.

pi@raspberrypi:~ $ sudo systemctl status homegenie.service ● homegenie.service - LSB: Run HomeGenie Loaded: loaded (/etc/init.d/homegenie; generated; vendor preset: enabled) Active: active (exited) since Sat 2019-02-02 12:21:02 CST; 3 days ago Docs: man:systemd-sysv-generator(8) Process: 5624 ExecStop=/etc/init.d/homegenie stop (code=exited, status=0/SUCCE Process: 6079 ExecStart=/etc/init.d/homegenie start (code=exited, status=0/SUC CGroup: /system.slice/homegenie.service

After restart found V.37 update, Installing and we'll see what happens again in 3 days Does anyone need / want ogging?

mralapete commented 5 years ago

Could you type mono —version at the terminal and paste the results here.

Have you sudo apt update and sudo apt upgrade recently. It may be worth doing to get the lastest version of your OS

Could you run HG in debug as follows

cd /usr/local/bin/homegenie sudo mono HomeGenie.exe

and paste the results.

tuicemen commented 5 years ago

Does anyone need / want ogging?

Logging is helpful in situations like this it can point to the root cause of the issue. If you know what to look for. This is why @mralapete & I asked what the logging showed and why @mralapete is asking you to run in debug mode and post the log.

Logging doesn't need to be running 24/7 unless your having problems. If the log file in HG shows nothing then a Debug logging using the command @mralapete posted usually will help.

ngineerken commented 5 years ago

On Feb 6, 2019, at 4:55 AM, mralapete notifications@github.com wrote:

Could you type mono —version at the terminal and paste the results here.

pi@raspberrypi:~ $ mono --version Mono JIT compiler version 4.6.2 (Debian 4.6.2.7+dfsg-1) Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com TLS: normal SIGSEGV: normal Notifications: epoll Architecture: armel,vfp+hard Disabled: none Misc: softdebug LLVM: supported, not enabled. GC: sgen

Have you sudo apt update and sudo apt upgrade recently. It may be worth doing to get the lastest version of your OS

I run this 2 to 3 times per week: pi@raspberrypi:~ $ sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade Could you run HG in debug as follows

cd /usr/local/bin/homegenie sudo mono HomeGenie.exe

and paste the results.

pi@raspberrypi:/usr/local/bin/homegenie $ sudo mono HomeGenie.exe rmmod: ERROR: Module lirc_atiusb is not currently loaded rmmod: ERROR: Module ati_remote is not currently loaded rmmod: ERROR: Module rc_ati_x10 is builtin. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/genielabs/HomeGenie/issues/361#issuecomment-460981626, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUc6Y5EEYax6K3swgPVaLYAnsi6mhS7ks5vKrS7gaJpZM4adqUH.

tuicemen commented 5 years ago

@ngineerken do you have HG password protected? You really should get more info then that.

mralapete commented 5 years ago

Those errors relate to the blacklisting of drivers which were conflicting with the X10 CM19 driver. Not sure those rmmod commands are required anymore.

I'm assuming you had logging enabled and examined the output in /usr/local/bin/homegenie/log/

To examine the last part of the log file before it hung just use the tail -n 20 homegenie.log and paste the results. This might catch the offending process which is causing system hang.

I assume you're running the WeeWX on the same Pi. Is it still running after HG has hung. Is it feeding from your own weather station or from the likes WU. I imagine it's quite resource hungry as its constantly logging.

I noticed you're using a PiDrive too. It might be worth swapping back to an SD card temporarily and run your setup to see if it still crashes.

Would you consider off loading WeeWX to a separate Pi and passing whatever you need from it to the Pi running HG.

ngineerken commented 5 years ago

On Feb 6, 2019, at 4:55 AM, mralapete notifications@github.com wrote:

Could you type mono —version at the terminal and paste the results here.

Have you sudo apt update and sudo apt upgrade recently. It may be worth doing to get the lastest version of your OS

Could you run HG in debug as follows

cd /usr/local/bin/homegenie sudo mono HomeGenie.exe

and paste the results.

pi@raspberrypi:~ $ cd /usr/local/bin/homegenie/log/ pi@raspberrypi:/usr/local/bin/homegenie/log $ tail -n 20 homegenie.log 2019-02-02 09:58:16.8539 Error HomeAutomation.HomeGenie.Automation 93 Method 'string.Split' not found. Exception.StackTrace at HomeGenie.Automation.ProgramManager.RoutePropertyChangedEvent (System.Object eventData) [0x000c3] in :0 Trace Complete 2019-02-02 09:58:16.7478 Trace [[[ EN2019-02-02 09:58:17.3491 Trace WaitAck 2019-02-02 09:58:17.3491 Trace Sending Message (Node=12, CallbackId=9E, Function=SendData, CommandClass=Version) 2019-02-02 09:58:17.3499 Debug 01-09-00-13-0C-02-86-11-05-9E-E7 2019-02-02 09:58:17.3514 Debug 06 2019-02-02 09:58:17.4517 Debug 01-04-01-13-01-E8 2019-02-02 09:58:17.4522 Debug ZWaveMessage (RawData=01-04-01-13-01-E8) 2019-02-02 09:58:17.4522 Debug ZWaveMessageessage (RawData=01-04-01-13-01-E8) Updated (NodeId=12, Parameter=ManufacturerSpecific, Value=ZWaveLib.CommandClasses.ManufacturerSpecificInfo) 19-02-02 09:58:15.5448 Trace SendDa2019-02-02 09:58:16.8507 Info HomeAutomation.ZWave 12 ZWave Node ZWaveNode.ManufacturerSpecific 0060:000C:0001 2019-02-02 09:58:16.8539 Error HomeAutomation.HomeGenie.Automation 93 Method 'string.Split' not found. Exception.StackTrace at HomeGenie.Automation.ProgramManager.RoutePropertyChangedEvent (System.Object eventData) [0x000c3] in :0 Trace Complete 2019-02-02 09:58:16.7478 Trace [[[ EN2019-02-02 09:58:17.3491 Trace WaitAck 2019-02-02 09:58:17.3491 Trace Sending Message (Node=12, CallbackId=9E, Function=SendData, CommandClass=Version) 2019-02-02 09:58:17.3499 Debug 01-09-00-13-0C-02-86-11-05-9E-E7 2019-02-02 09:58:17.3514 Debug 06 2019-02-02 09:58:17.4517 Debug 01-04-01-13-01-E8 2019-02-02 09:58:17.4522 Debug ZWaveMessage (RawData=01-04-01-13-01-E8) 2019-02-02 09:58:17.4522 Debug ZWaveMessagepi@raspberrypi:/usr/local/bin/homegenie/log $

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/genielabs/HomeGenie/issues/361#issuecomment-460981626, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUc6Y5EEYax6K3swgPVaLYAnsi6mhS7ks5vKrS7gaJpZM4adqUH.

ngineerken commented 5 years ago

On Feb 7, 2019, at 11:39 AM, mralapete notifications@github.com wrote:

Those errors relate to the blacklisting of drivers which were conflicting with the X10 CM19 driver. Not sure those rmmod commands are required anymore.

I'm assuming you had logging enabled and examined the output in /usr/local/bin/homegenie/log/

To examine the last part of the log file before it hung just use the tail -n 20 homegenie.log and paste the results. This might catch the offending process which is causing system hang.

I assume you're running the WeeWX on the same Pi. Is it still running after HG has hung. Is it feeding from your own weather station or from the likes WU. I imagine it's quite resource hungry as its constantly logging.

I noticed you're using a PiDrive too. It might be worth swapping back to an SD card temporarily and run your setup to see if it still crashes.

Would you consider off loading WeeWX to a separate Pi and passing whatever you need from it to the Pi running HG.

If I have to split the weather station & Home Genie I might consider another Home Automation. But the ones that I have experience in the past were clunky as compared to HG. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/genielabs/HomeGenie/issues/361#issuecomment-461525463, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUc6f6Jt9EZ5i_7UhppKzxuvqwvZR8Sks5vLGS_gaJpZM4adqUH.

genemars commented 5 years ago

@ngineerken from your logs it seems you're running an old versiono of mono ad I see "Method 'string.Split' not found". Mono 4.6.2 is recommended.

genemars commented 5 years ago

@genemars ... ok just noticed from your previous message that you're already running mono 4.6.2. So this error is very odd.

genemars commented 5 years ago

@ngineerken did you try also latest HomeGenie release?

ngineerken commented 5 years ago

On Feb 9, 2019, at 10:15 AM, Generoso Martello notifications@github.com wrote:

@ngineerken https://github.com/ngineerken did you try also latest HomeGenie release?

I have HomeGenie Version v1.2-stable.37 installed — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/genielabs/HomeGenie/issues/361#issuecomment-462057183, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUc6abSKUfQscVbVENbukxkxCLe0SZeks5vLvQagaJpZM4adqUH.

tuicemen commented 5 years ago

@ngineerken are you still experiencing this?

ngineerken commented 5 years ago

Yes it is till happening and even more frequent like ever 2 days now and it is with HG 1.2.stable 39

tuicemen commented 5 years ago

Does weewx also stop? Have you tried running just HG or weewx on the pi and the other on another machine (pi)?

ngineerken commented 5 years ago

Weewx does not stop. I have not had a chance to try another pi at this time. I may try it this weekend, just running from an SD card.

genemars commented 5 years ago

Hi @ngineerken can you try disabling the "Devices & Things" -> "UPnP Emulation" program?

ngineerken commented 5 years ago

Started the SD card on a new Raspberry pi 3B+ now i have the Beta offering multiple installs https://github.com/genielabs/HomeGenie/issues/373

tuicemen commented 5 years ago

@ngineerken did disabling the "UPnP emulation" program fix your HG hang ups?

ngineerken commented 5 years ago

I have HomeGenie Version v1.3-beta.2 running on a new SD card on a Pi3b+, and is running fine so far. Yes I have UPnP disabled too.

tuicemen commented 5 years ago

Gene did state a bug in the UPnP program was causing HG to hang in another thread. I had to disable that as well but for a different reason. I think it was so the echo bridge would work so that maybe why I didn't get the issue you had. Nice to see it has been working for you with out hanging so far.

noliono commented 5 years ago

I have HomeGenie Version v1.3-beta.2 running on a new SD card on a Pi3b+, and is running fine so far. Yes I have UPnP disabled too.

So issue can be clode no? @ngineerken Resolve by deactivating emulation upnp..

genemars commented 5 years ago

I think this can be closed now.