Plugin needs improved connection error handling

[gt3mike]

This has happened several times in the last couple of days. Some sort of network issue causes a plugin connection failure, and instead of self recovering, the plugin goes into a loop with an error every ten minutes. Restarting the plugin from the Indigo console is successful every time.

Could this kind of connection failure be caught and handled with functionality being eventually restored once the network is running again? The ten minute loop seems to imply to me that the retry logic is already there, but there may be an issue with what the retry is attempting to do.

Here's the log.

Dec 6, 2020 at 1:50:54 AM
   Lutron RRA2/Caséta Error       Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 1053, in runConcurrentThread
  File "plugin.py", line 158, in poll
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/telnetlib.py", line 430, in read_eager
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/telnetlib.py", line 576, in fill_rawq
error: [Errno 60] Operation timed out

   Lutron RRA2/Caséta Error       plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
   Lutron RRA2/Caséta Error       Lutron IP Gateway: EOFError: telnet connection closed
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

**(Snip. There are a lot more of these.)**

   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 1:57:43 AM
   Error                           email scan timeout  **(Obviously there was some kind of network error happening)**

Dec 6, 2020 at 2:00:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 2:10:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 2:20:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 2:30:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 2:40:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 2:50:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 3:00:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

(Snip.)

Dec 6, 2020 at 8:20:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 8:30:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 8:40:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 8:50:00 AM
   Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Dec 6, 2020 at 8:54:40 AM
   WebServer                       Indigo Touch client connected from 192.168.0.221

Dec 6, 2020 at 8:56:39 AM
   Reloading plugin "Lutron RRA2/Caséta 7.3.3"
   Stopping plugin "Lutron RRA2/Caséta 7.3.3" (pid 10041)
   Lutron RRA2/Caséta             Shutting down Lutron
   Stopped plugin "Lutron RRA2/Caséta 7.3.3"
   Starting plugin "Lutron RRA2/Caséta 7.3.3" (pid 14790)
   Started plugin "Lutron RRA2/Caséta 7.3.3"
   Lutron RRA2/Caséta             Starting up Lutron
   Lutron RRA2/Caséta             Lutron IP Gateway: Running IP Start
   Lutron RRA2/Caséta             Lutron IP Gateway: Connecting via IP to 192.168.0.4:23

[FlyingDiver]

Yeah, I worked on the reconnect code some, but I need to take another pass at it. Hard for me to test without shutting down the system here, which annoys the spouse. ;)

[gt3mike]

The good news is, the retry logic seems solid. :) Seems like something is getting broken when the connection initially fails and it isn't being fully reset.

[rapamatic]

I have a similar issue sporadically. Anything specific from the logs that could help diagnose the issue? I just assume my ancient Mac Pro has random connectivity issues.

[FlyingDiver]

I can't help with the original disconnects, that's an OS or network issue. I think. But I should be able to get it to reconnect more reliably. I just need to figure out how to close out everything and start from scratch. Or if it's a problem that the process is messed up, just restart the plugin. Which might be the right answer.

[gt3mike]

Yep, I think my router, switches, or access points might have received a firmware update. Nothing you can do about that. Restarting the plug-in seems like a pretty big hammer if it is going to restart with every failed retry. But if that’s what it takes, that’s what it takes I guess.

[FlyingDiver]

Fixed in next release. Tested with two scenarios. First, network disconnect by unplugging the ethernet to the repeater. Second, power failure by unplugging power to repeater. In both cases, the plugin reconnected to the repeater properly within 30 seconds of network or power restored.

[gt3mike]

Were you able to consider a temporary network failure? You could emulate it by disconnecting ethernet from the host Mac.

[FlyingDiver]

No difference. Unplug the cable at the repeater or unplug it at the Indigo Mac. Same symptoms, same solution.