Retrieving the ALDB (API question)

I'm attempting to write my own little tool for doing some maintenance of the ALDB, starting with the PLM. It's not going super, and I'm looking for some help. Based on that help, I'll try to contribute to the documentation and make it easier to understand.

What I'm doing involves Qt so I create the asyncio loop using quamash. I don't think that affects my question, but I'm including this for completeness, as I'm a Java/Scala/C/C++ developer hacking my way through Python-land:


loop = QEventLoop(app)
print(loop)
asyncio.set_event_loop(loop)```

My attempt to dump the ALDB starts with just printing to the console:

```class Controller:
  def __init__(self, loop):
    self.loop=loop
    print("Controller initialized")

  def connect(self):
    self.loop.create_task(self._connect())

  def load(self):
    self.loop.create_task(self._load())

  def dump(self):
     self.loop.create_task(self._dump())

  @asyncio.coroutine
  def _connect(self):
    self.conn = yield from insteonplm.Connection.create();
    print("Connected")

  @asyncio.coroutine
  def _load(self):
    print("Loading...")
    self.plm = self.conn.protocol
    self.plm.aldb.clear()
    yield from asyncio.sleep(1, loop=self.loop)
    self.plm.read_aldb()

    yield from asyncio.sleep(5, loop=self.loop)

    while self.plm.aldb.status == ALDBStatus.LOADING:
      print(str(self.plm.aldb.status))
      yield from asyncio.sleep(1, loop=self.loop)

    print(str(self.plm.aldb.status))
    print("Done.");

  @asyncio.coroutine
  def _dump(self):
    for i in self.plm.aldb:
      record=self.plm.aldb[i]
      print(record)```

The problem is I get very few records - sometimes zero, sometimes two.  There should be something on the order of 80 links.  This is all with 0.13.1.

Am I doing something obviously wrong here?

I figured it out. It was polling the aldb by itself as soon as I connected. I really don't want it to do that, at least not unless it can somehow give me a callback telling me when it's done ....

It does NOT set self.plm.aldb.status while it is doing this - it shows ALDBStatus.EMPTY the whole time.

Class PLM is a subclass of IM which is a subclass of Device. PLM has a constructor parameter load_aldb that defaults to True. But it doesn't appear to be used anywhere.

In /init.py I changed Connection.create as follows::

    @asyncio.coroutine
    def create(cls, device='/dev/ttyUSB0', host=None,
               username=None, password=None, port=25010,
               auto_reconnect=True, loop=None, workdir=None,
               poll_devices=True, load_aldb=True):```

then changed how PLM is created:

    protocol_class = PLM
    if conn.host:
        protocol_class = Hub
    conn.protocol = protocol_class(
        connection_lost_callback=connection_lost,
        loop=conn.loop,
        workdir=workdir,
        poll_devices=poll_devices,
        load_aldb=load_aldb)```

which in theory should let me change the way I create the connection. The problem is, Device doesn't actually use this parameter. I would add it, but I can't really figure out what in this whole process triggers it to fetch the ALDB.

Another promising thought was to use this from Device:

self._all_link_complete_callback = None

but it does not appear that is used anywhere, either.

So something has to happen here to fix this. I'm willing to put in the work but I need some guidance. Options:

Figure out how to propagate the load_aldb option to Device and make it so that I can actually prevent the initial load.
Make it so that when whatever is triggering this starts it sets the state to LOADING so I can observe it with polling and a timer (I'm not a fan of this, really)
Finish the callback thought and give me a way to find such a thing from the PLM, IM, or Device object, and hook into it. This isn't great either, though, because it's a race condition.

The real problem with the initial load of the aldb, as soon as I create the Connection, is that I have no idea when it's complete. I'm trying to drive a UI here so I'd really prefer some kind of event-driven way to tell. I usually load it as the first thing but I don't always want to, and when I do, I need to know when it's done.

Suggestions?

Working my way through this, I've figured out a few things. One is that there is a way to know when the initial loading is done:


  @asyncio.coroutine
  def _connect(self):
    self.conn = yield from insteonplm.Connection.create(device="/dev/ttyUSB0");
    self.plm = self.conn.protocol
    self.plm.add_all_link_done_callback( self.aldb_loaded_callback )

  def aldb_loaded_callback(self):
    print("Initial all-link database now loaded")```

I still think it should update the ALDB status to LOADING while it is doing this, but it doesn't.  I think some of this confusion is because how it works: when loading an ALDB from the PLM you use PLM link layer command 0x69.

I also discovered that there is a quick_start option that I think overlaps with what `load_aldb` is supposed to do, but similar to the latter, it is only partially implemented.  I fixed these, then tried creating a connection without loading the ALDB, then calling `plm.read_aldb()` later.  This failed, because it decided to do so using an insteon message (plm Send Extended, 0x62, with cmd1=0x2f). 

So I THINK the solution here is to make method `_refresh_aldb()` in class IM explicitly public, then everything should work - you have the option of creating a Connection to the PLM with our without loading the aldb, and you can load it (or reload it) later as you wish.

Thoughts?

I have started working on some changes to the ALDB which will allow for direct writes to the modem. The ALDB of the modem works different than the ALDB of other devices. This is, in my opinion, a problem with Insteon that it does not have a meaningful API that allows it to keep DB changes in sync and it does not have a real method to ensure you don't break things when you write to the ALDB.

Bottom line, writing to he ALDB is a high risk situation and requires a lot of testing. I am also good to remove the ALDB load function on startup and make that a function of the front end. This only means that the consumer of the application will need to know how to load devices into the Modem.devices list.

The current method is kind of an idiot proof method of ensuring any device in the devices list is in the modem's database. If you add a device to the devices list that is not in the modem's database then you will get inconsistent results. This is a limitation of Insteon, not the library.

Just to explain what I'm trying to do here ...

I really want the ability to capture the entire ALDB (from the modem) to a readable file (JSON, XML, CSV, whatever) ... be able to edit it to fix/remove bogus stuff ... then replace the whole thing. To do this, you'd have to write any record from the NEW list not in the IM, then delete any record in the IM that's not in the NEW list. But alternatively you could just wipe out the entire ALDB on the IM and start over. That would take longer, I think, but maybe not if you can accomplish this by just setting the high watermark on the first record - I'm not sure. Whatever's safer.

The field devices are trickier because there are direct links (switches to loads for example) that don't involve the IM.

I really want to have the ability to write some kind of lint checker for these links. I want for example to be able to find half-links; find things in the IMDB that aren't in the field devices; and detect things like a field device has a direct link to an appliance module that's no longer on the network (which slows/chokes things like crazy). I'd like to never have to run HouseLinc again (how I currently do all this stuff).

So I look forward to seeing where you're going with this.

@wz2b Hope you are well. Not sure if you have been following pyinsteon but I have made a lot of progress. Here is the underlying architecture. All layers communicate via a pub/sub model. I like the new architecture much better. I appreciate the feedback regarding the complexity of the insteonplm object model.

Hi Tom!

@wz2b https://github.com/wz2b Hope you are well. Not sure if you have been following pyinsteon but I have made a lot of progress. Here is the underlying architecture. All layers communicate via a pub/sub model. I like the new architecture much better. I appreciate the feedback regarding the complexity of the insteonplm object model.

So it's funny, you sent this e-mail while I was in Las Vegas, so I marked it to respond after I returned and for some odd reason it ended up in the wrong place and never notified me. This morning I woke up and it popped into my head that I hadn't gotten back to you. Strange.

I haven't been paying attention. I have been off in the world of golang, and my organization got a big contract for the government (having to do with large vehicle electrification) that has occupied me. But I am still interested. I just need to get back into the groove. The second odd coincidence here is that I had a keypadlinc fail on me. I replaced it, but now I'm trying to figure out the least painful way to re-link everything, since it has a lot of buttons that have to be linked individually. So what you are writing about here is completely on topic. The main thing I'm really looking for, still, is

Capture the ALDB of the entire network (every device) to a file
Be able to edit it by hand etc
Be able to tell some piece of software "ok, now go put it all back / full synchronization / delete records that shouldn't be there" .. being able to sync just one device is good too but if you're linking usually there's at least 2 things needing updating

So in my case, at least, a fancy GUI to do all this isn't even necessary, as long as there's a file I can just edit somewhere. That file can be pretty much anything, though - including python.

So that's a long-winded way of me saying I'll go download / update everything today and try to catch myself back up to where you are, then figure out how I can contribute, either in software, thought, or at least just helping test.

Thanks very much for reaching out. Sorry I disappeared for so long.

-C

On Sun, Jan 19, 2020 at 7:33 PM Tom Harris notifications@github.com wrote:

@wz2b https://github.com/wz2b Hope you are well. Not sure if you have been following pyinsteon but I have made a lot of progress. Here is the underlying architecture. All layers communicate via a pub/sub model. I like the new architecture much better. I appreciate the feedback regarding the complexity of the insteonplm object model. [image: Architecture] https://user-images.githubusercontent.com/9055260/72691329-879e0c00-3af2-11ea-9211-f1c19bfdeb8e.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nugget/python-insteonplm/issues/99?email_source=notifications&email_token=AALTMOICJSMMCX6Y5AEZJOTQ6TWNFA5CNFSM4FTHJIGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJLBBHQ#issuecomment-576065694, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALTMOMJ2QPQBTVDPK6RANDQ6TWNFANCNFSM4FTHJIGA .

nugget / python-insteonplm

Retrieving the ALDB (API question) #99