Crash after extended run

[sabolcik]

Hey Kevin:

I have been running for longer periods of time on my logger as some of the non-Vera devices that were failing have been corrected.

I had a multiday run going then received an error:

Exception in thread Thread-848:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/vera_connect.py", line 86, in run_poll
    data = json.loads(connect())
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/vera_connect.py", line 84, in connect
    return connect()
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/vera_connect.py", line 84, in connect
    return connect()
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/vera_connect.py", line 84, in connect
    return connect()
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/vera_connect.py", line 84, in connect
    return connect()

These connect errors repeat go on for an extended number of lines before I get this different part of the exception dump:

 File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/vera_connect.py", line 79, in connect
    timeout=1
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 57, in request
    with sessions.Session() as session:
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 392, in __init__
    self.mount('https://', HTTPAdapter())
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 126, in __init__
    self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 162, in init_poolmanager
    block=block, strict=True, **pool_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 155, in __init__
    dispose_func=lambda p: p.close())
  File "/usr/local/lib/python2.7/dist-packages/urllib3/_collections.py", line 47, in __init__
    self._container = self.ContainerCls()
  File "/usr/lib/python2.7/collections.py", line 69, in __init__
    self.__update(*args, **kwds)
  File "/usr/lib/python2.7/abc.py", line 151, in __subclasscheck__
    if subclass in cls._abc_cache:
  File "/usr/lib/python2.7/_weakrefset.py", line 75, in __contains__
    return wr in self.data
RuntimeError: maximum recursion depth exceeded in cmp

My database thread was still running and my other loggers were still logging away. I will try to dig into the vera_connect file to see if anything makes sense. I am not wrapping every vera access with a try/except clause but will attempt to put some additional debugging into the threads to see if I can get better visibility. Based on the logging that I already have I think this error occurred in the vera callback thread. This is the only place that I access the vera library after the initial call.

Does this exception indicate anything obvious to you?

[sabolcik]

One additional datapoint. I was reviewing the code in vera_connect that failed and it looks like you're just trying to send a JSON to the vera and it times out.

Here is the code I am running in my vera event callback:

def device_callback(event):

    update_lock.acquire()
    now = datetime.datetime.now().isoformat()

    name = event.name
    split_event = event.event.split('.')

    if split_event[-1] == 'changed':    
        parameter_name = split_event[-2:][0]

        try:
            vera_log.update_value(name, parameter_name)
            logging.info('--------------------------------------------')
            logging.info(now)
            logging.info("Callback: " + event.event)
            logging.info(name)
            logging.info(parameter_name)
            logging.info("Value: " + vera_log.get_value(name, parameter_name))
            logging.info('--------------------------------------------')
            json = vera_log.get_json()
            update_database(db_client, json)
        except KeyError:
            logging.info("Invalid device: " + name + " " + parameter_name)

    update_lock.release()

The vera_log.update_value is a simple class I have the maintains some data to make it easy to write to the database. It basically just calls getattr on the vera instance for the particular device that created the callback.

I added the following additional exception handler after the KeyError exception:

except:
            logging.info("Error logging Vera sample in device_callback")
                        traceback.print_exc()

Not sure the library will recovery gracefully if I simply ignore this failure but I will run it and give it a shot.

[kdschlosser]

yes that error does make sense to me, It could be that your Vera took a long time to reboot. or the lua engine froze up on it. some kind of a network issue. I do know that at say like 2:00 AM every day the Vera does a zwave heal. and when doing this it shuts down all other communications. it could have taken to long and you banged against the recursion depth limit in python. instead of making a looping call to connect I will use a while loop. and i will also generate a disconnected and connected event

[kdschlosser]

I have updated the library. give it a go. I also added 2 events vera.connected and vera.disconnected this way you can have your program change what it is doing when this happens. if you are not running the polling loop the only way a disconnect/connect event will take place is when you change something in the library.

[sabolcik]

After reviewing the code it did appear that the event was a timeout with my Vera. Wasn't sure the best way to handle the timeout.

With your most recent change what would be the preferred way to handle a timeout or do I even need to worry about a timeout condition? I only read the Vera at initial startup and in the event handler. Does your change to a while loop mean that the event handler won't get called in the disconnected condition and hence my code will not get invoked in the event handler if the Vera is disconnected? That way I don't need to do anything special like checking the disconnect state.

Thanks again for the quick turnaround. I will download the updated code this evening.

Regards....

On Mon, Feb 19, 2018 at 12:01 PM, Kevin Schlosser notifications@github.com wrote:

I have updated the library. give it a go. I also added 2 events vera.connected and vera.disconnected this way you can have your program change what it is doing when this happens. if you are not running the polling loop the only way a disconnect/connect event will take place is when you change something in the library.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kdschlosser/MiCasaVerde_Vera/issues/5#issuecomment-366766447, or mute the thread https://github.com/notifications/unsubscribe-auth/AhSL40G-jvC-cfJ3-bBtVpI5QZQobmMIks5tWbcEgaJpZM4SJ0K_ .

[kdschlosser]

no events will take place when it is in a disconnected state. save for the event letting you know that it has become disconnected.

You can use the disconnected event to maybe give you some kind of a notification the thing has become disconnected.

The modifications I made does not recurse at all and will keep on looping until the end of time connection or no connection. it only produces a single event for connection and disconnection while in the loop. I will need to modify this so it will produce an event on poll startup.

I do have another library i keyed up that you can send SMS and MMS text messages with. I do not have it up on github i can put it up if ya like. this way you can get a text message sent to you phone if there is some kind of a connection issue.

[sabolcik]

Hi Kevin:

Thanks for the quick response. Let me get this current version up and running. I'll ping you to let you know if I want to go the full SMS route.

On Tue, Feb 20, 2018 at 4:59 PM, Kevin Schlosser notifications@github.com wrote:

no events will take place when it is in a disconnected state. save for the event letting you know that it has become disconnected.

You can use the disconnected event to maybe give you some kind of a notification the thing has become disconnected.

The modifications I made does not recurse at all and will keep on looping until the end of time connection or no connection. it only produces a single event for connection and disconnection while in the loop. I will need to modify this so it will produce an event on poll startup.

I do have another library i keyed up that you can send SMS and MMS text messages with. I do not have it up on github i can put it up if ya like. this way you can get a text message sent to you phone if there is some kind of a connection issue.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kdschlosser/MiCasaVerde_Vera/issues/5#issuecomment-367151186, or mute the thread https://github.com/notifications/unsubscribe-auth/AhSL499FSLvABFjy_JZqQRGNVI-yAxRJks5tW05cgaJpZM4SJ0K_ .

[sabolcik]

I was able to download the library and have gotten a few errors.

First in the scene constructor I have a parameter on my vera that wasn't in the standard scene constructor arg list:

Traceback (most recent call last):
  File "<stdin>", line 128, in <module>
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/__init__.py", line 332, in connect
    return __Vera(ip_address)
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/__init__.py", line 292, in __init__
    _VeraBase.__init__(self, ip)
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/__init__.py", line 425, in __init__
    self.__update(self.scenes, 'scenes', data)
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/__init__.py", line 477, in __update
    obj.update_node(data.pop(key, None), True)
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/scenes.py", line 112, in update_node
    found_scene = Scene(self, **scene)
TypeError: __init__() got an unexpected keyword argument 'paused'

I added the parameter paused to the scene constructor to get past that error. That got the library running but then I got this error:

Exception in thread Thread-848:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/vera_connect.py", line 85, in run_poll
    Notify('vera.disconnected', self)
  File "/home/sabolcik/.local/lib/python2.7/site-packages/micasaverde_vera/event.py", line 48, in notify
    if fnmatchcase(event.lower(), event_name):
AttributeError: 'VeraConnect' object has no attribute 'lower'

That error kills the vera callback events. Any ideas?

[sabolcik]

I was able to get it running by removing what I think is the statement that raises the event for connected and disconnected in vera_connect.py. I'm not sure why the connect and disconnect call events behaves differently than the others

def send(self, **params):
        if 'output_format' not in params:
            params['output_format'] = 'json'

        self._lock.acquire()

        try:
            response = requests.get(self.URL, params=params)
            if not self._connected:
                self._connected = True
                #Notify('vera.connected', self)
        except (ConnectionError, Timeout, ReadTimeout, ConnectTimeout):
            if self._connected:
                self._connected = False
                #Notify('vera.disconnected', self)
            time.sleep(random.randrange(1, 5) / 10)
        else:
            try:
                return json.loads(response.content)
            except ValueError:
                if 'ERROR' in response.content:
                    if 'No implementation' in response.content:
                        raise VeraNotImplementedError
                    if (
                        'Device does not handle service/action' in
                        response.content
                    ):
                        raise VeraUnsupportedByDevice
        finally:
            self._lock.release()

[sabolcik]

Also had to pull the Notify events out of the run_poll() function.

[kdschlosser]

it's because i am passing the parameters to Notify Backwards,.

swap the string literal with the self.

so

Notify(self, 'vera.disconnected')

would become

Notify('vera.disconnected', self)

[sabolcik]

Yup. That fixed it. Thanks will keep you posted on any additional findings.

On Wed, Feb 21, 2018 at 11:39 PM, Kevin Schlosser notifications@github.com wrote:

it's because i am passing the parameters to Notify Backwards,.

swap the string literal with the self.

so

Notify(self, 'vera.disconnected')

would become

Notify('vera.disconnected', self)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kdschlosser/MiCasaVerde_Vera/issues/5#issuecomment-367573406, or mute the thread https://github.com/notifications/unsubscribe-auth/AhSL4zql9R2WbnadPgRksxohwZNwtB7-ks5tXP2sgaJpZM4SJ0K_ .

[kdschlosser]

I didn't know if you are still using this library or not. and if you are I just updated it. It has better output handling (using the logging module) and is also python 3.3+ compatible. I already made the changes posted here. I was also read over your other issue again. and I was thinking to myself.. I am such a ding dong.. this is an easy thing to code up.

# vera connection assumed

# start the polling loop. this is the speed in which the micasaverde_vera 
# library is going to update it's data from the vera. every 10 seconds.
# you can set this to 0.1 without causing too much of a performance impact on the vera
vera.start_polling(10.0)

import threading
event = threading.Event()

def loop():
    while not event.is_set():
        for device in vera.devices:
            watts = getattr(device, 'watts', None)
            if watts is None:
                watts = getattr(device, 'Watts')
            if watts is not None:
                # do your code here to handle the watts
                pass
        # 60 second stall between loops
        event.wait(60.0)

t = threading.Thread(target=loop)

# make sure the thread gets closed if the app crashes for some reason.
t.daemon = True

# start the thread
t.start()

# wait forever or until a keyboard interrupt occurs
try:    
    event.wait()
except KeyboardInterrupt:
    # set the event so the loop closes gracefully
    event.set()
    # join the thread if it is still running so it can finish up 
    # what it is doing and the process can close gracefully
    if t.is_alive():
        t.join()

vera.disconnect()

[sabolcik]

Hey Kevin:

Thanks for the update. I literally just got back to using the library again. I updated to a VeraPlus from my old VeraLite so that took a fair amount of monkeying to get it right. I'm hoping that the VeraPlus is more robust than the VeraLite.

I have a pretty complicated logging system with a bunch of different data sources feeding my database in addition to the Vera. I have separate loggers for my whole house energy monitor, my pool pump, my outdoor weather station, etc. They are all derived from a single logger class with different guts underneath.

I have an issue where every few weeks the system will crash. Ultimately I wanted to setup a cron job to restart my logger every evening at midnight to handle the hanging situation. Unfortunately I couldn't get a shell script that would fire off from cron to work properly so I never got around to it. I'm hoping that with some additional tweaking I can make that work.

I'll try installing your latest library in the coming days and see how it works for me.

Thanks again for putting such a great tool together.

Regards,

Ross

On Tue, Sep 10, 2019 at 3:22 PM Kevin Schlosser notifications@github.com wrote:

I didn't know if you are still using this library or not. and if you are I just updated it. It has better output handling (using the logging module) and is also python 3.3+ compatible. I already made the changes posted here. I was also read over your other issue again. and I was thinking to myself.. I am such a ding dong.. this is an easy thing to code up.

vera connection assumed

start the polling loop. this is the speed in which the micasaverde_vera # library is going to update it's data from the vera. every 10 seconds.# you can set this to 0.1 without causing too much of a performance impact on the vera

vera.start_polling(10.0) import threading event = threading.Event() def loop(): while not event.is_set(): for device in vera.devices: watts = getattr(device, 'watts', None) if watts is None: watts = getattr(device, 'Watts') if watts is not None:

do your code here to handle the watts

            pass
    # 60 second stall between loops
    event.wait(60.0)

t = threading.Thread(target=loop)

make sure the thread gets closed if the app crashes for some reason.

t.daemon = True

start the thread

t.start()

wait forever or until a keyboard interrupt occurstry:

event.wait()except KeyboardInterrupt:
# set the event so the loop closes gracefully
event.set()
# join the thread if it is still running so it can finish up
# what it is doing and the process can close gracefully
if t.is_alive():
    t.join()

vera.disconnect()

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kdschlosser/MiCasaVerde_Vera/issues/5?email_source=notifications&email_token=AIKIXY4QPADQWRDUXD5NVSTQI76YHA5CNFSM4ERHIK72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6ML22I#issuecomment-530103657, or mute the thread https://github.com/notifications/unsubscribe-auth/AIKIXY6SLJGZG3XDHFOVPOTQI76YHANCNFSM4ERHIK7Q .

[kdschlosser]

well how about this. I can add the bits needed so that this thing can run as a service and you can connect to it from a loopback socket connection or if you want to do up a remote connection I can do that as well and secure it with AES and optionally SSL.

The whole system will operate the same for the most part. the only things that would be added that would be different is on the client side you point the library at the "server" service instead of a Vera and then you supply the password to connect to the service and optionally the path to the ssl cert files.

the service will just sit there in a loop receiving updates from the Vera and updating whatever it needs to and it will wait for a client to ask for information.

The funny thing is not a few weeks ago i just coded up this exact same kind of a connection scheme for python-openzwave so it can be installed as a service on Windows, Linux and OSX, it to also allows for remote connections. So it shouldn't be all to hard to do some copying and pasting.

Do NOT test the new version of the library in your production environment. I am not sure if it is running properly and I do not want you to end up with any corrupted data in your database. The Vera I am not worried about because it's not going to change anything on it. I do not know if it is going to build the files correctly. There is another user that posted an issue about it not running in python 3. so I made the changes needed to get it running in python 3. He is still having problem getting it to run. I no longer have a Vera (YAY!!) and I am pretty much heading up python-openzwave now. so my knowledge of the low level portion of ZWave is pretty good.

I do not remember how to get this library running anymore. so if you can chime in on the issue #8 the user seems like they might be newish to python not sure tho. maybe you can walk them through how to connect the library to the vera.

[sabolcik]

Hey Kevin:

Thanks for the update. I'd be game to try the Unix service but would likely need some sample code showing how to interface with it. It's been a long time since I've done anything with services. So if you're game I'm up for it.

I can check out the issue on github and see if I can remember the steps I went through to get everything working. Surprisingly moving to the new Vera wasn't too bad with my system. I needed to do some configuration on the devices on my Vera so they would report wattage properly but the vera python code worked without a hitch.

Interesting that you coded the OpenZwave library. I run 1/2 of the IoT business at Silicon Labs and we purchased ZWave last year. It's a great technology and gives us pretty much every wireless portfolio under one roof.

Regards,

Ross

On Thu, Sep 19, 2019 at 2:02 AM Kevin Schlosser notifications@github.com wrote:

well how about this. I can add the bits needed so that this thing can run as a service and you can connect to it from a loopback socket connection or if you want to do up a remote connection I can do that as well and secure it with AES and optionally SSL.

The whole system will operate the same for the most part. the only things that would be added that would be different is on the client side you point the library at the "server" service instead of a Vera and then you supply the password to connect to the service and optionally the path to the ssl cert files.

the service will just sit there in a loop receiving updates from the Vera and updating whatever it needs to and it will wait for a client to ask for information.

The funny thing is not a few weeks ago i just coded up this exact same kind of a connection scheme for python-openzwave so it can be installed as a service on Windows, Linux and OSX, it to also allows for remote connections. So it shouldn't be all to hard to do some copying and pasting.

Do NOT test the new version of the library in your production environment. I am not sure if it is running properly and I do not want you to end up with any corrupted data in your database. The Vera I am not worried about because it's not going to change anything on it. I do not know if it is going to build the files correctly. There is another user that posted an issue about it not running in python 3. so I made the changes needed to get it running in python 3. He is still having problem getting it to run. I no longer have a Vera (YAY!!) and I am pretty much heading up python-openzwave now. so my knowledge of the low level portion of ZWave is pretty good.

I do not remember how to get this library running anymore. so if you can chime in on the issue #8 https://github.com/kdschlosser/MiCasaVerde_Vera/issues/8 the user seems like they might be newish to python not sure tho. maybe you can walk them through how to connect the library to the vera.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kdschlosser/MiCasaVerde_Vera/issues/5?email_source=notifications&email_token=AIKIXY5SL3SRY2Y3KI2CMH3QKMPXTA5CNFSM4ERHIK72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7COEJY#issuecomment-532996647, or mute the thread https://github.com/notifications/unsubscribe-auth/AIKIXYZL7G64BFNMKBBAVNLQKMPXTANCNFSM4ERHIK7Q .

[kdschlosser]

I do not code OpenZWave.. I do code python-openzwave. python-openzwave is a python binding for OpenZWave. there is all kinds of code that has to be written to handle the interface between Python code and C++ code. and also an Python API to know what to do with the data from OpenZWave.

I actually just asked sent in my first support request to Sil about the ZWave Specification. and there was an error or an inconsistency between the supplied header file and what the written specification was stating... I am also trying to get a better understabnding of how the THERMOSTAT_FAN_STATE works in terms of knowing what data a node can send and how I would go about asking the node for what it supports in terms of the state... I am waiting for an answer back on that last part..