search

openhab / openhab-addons

Add-ons for openHAB
https://www.openhab.org/
Eclipse Public License 2.0
1.9k stars 3.59k forks source link

[hueemulation] Re-sequences HueId's on openHAB restart - out-of-sync w/ Alexa #4602

Closed mrguessed closed 5 years ago

mrguessed commented 5 years ago

Expected Behavior

The MAP of Hue Id's <-> openHAB Item names should remain constant throughout the lifetime of the Hue Emulation service.

Any Hue Id allocated by the service should not be re-used for a different Item name, and must survive openHAB Process restarts.

Current Behavior

The Hue Emulation service stores the MAP of Hue Id's to openHAB Item names is kept in JSONDB.

Under Linux, this ends up as JSON Text in a file like:

/var/lib/openhab2/jsondb/hue.emulation.lights.json

with contents like:

{
  "GManagedFamilyBlackout": {
    "class": "java.lang.Integer",
    "value": 11
  },
  "GManagedHolidaySwitch": {
    "class": "java.lang.Integer",
    "value": 21
  },
  ...
}

Hue Id's within this file are allocated sequentially, by the Hue Emulation service, and the file is written out as new entries are discovered.

This file is also read at Hue service (and thus openHAB) startup.

Hue Id's collected in this file are (indirectly) advertised to the Amazon Alexa service, during it's "Discover Devices" command, in the form of Hue Emulation Service URL's to call to interact with the Item.

When openHAB is shutdown, this file is cleared out and the original MAP of Hue Id's is lost. Upon openHAB restart, the file/storage is again rebuilt from the current version of the Items in the system.

If the Item definitions are changed (eg. Lighting tags added/removed), then the rebuilt MAP will now be inconsistent with that advertised to the Amazon Alexa service during it's "Discover Devices" command.

In my case, my Item declarations are in .items files, and I tag/add/remove items within these files.

In my case, I have 22 Items using Hue Emulation service.

harmony.items:Switch FamilyHarmony_tv    "TV"     ["Switchable"]
harmony.items:Switch FamilyHarmony_dvd   "DVD"    ["Switchable"]
house.items:Group:Switch:OR(ON,OFF) GManagedGoodnight "House" (GManaged) ["Switchable"]
house.items:Group:Switch:OR(ON,OFF) GManagedBreakfast "Breakfast" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedDinner "Dinner" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedBlackout "Blackout" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedBlinds "Blinds" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedFamilyBlackout "Family Blackout" (GManagedBlackout) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedFamilyBlinds "Family Blinds" (GManagedBlinds) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedKitchenBlackout "Kitchen Blackout" (GManagedBlackout) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedKitchenBlinds "Kitchen Blinds" (GManagedBlinds) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedFamily "Family" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedKitchen "Kitchen" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedLiving "Living" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedMaster "Master" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedDining "Dining" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedHolidaySwitch "Holiday" (GManaged) ["Lighting"]
house.items:Group:Switch:OR(ON,OFF) GManagedMediaLights "Media" (GManaged) ["Lighting"]
house.items:Switch   KitchenIslandLightsStatus "Kitchen Lights" (GSwitch,GManagedKitchen,GManagedEmergency,GManagedGoodnight,GManagedBreakfast,GManagedSwitch) ["Lighting"] {mios="unit:house,device:27/service/SwitchPower1/Status",mqtt=">[home:house/controller/kitchen_island_lights:state:*:MAP(miosSwitchOut.map)]"}
house.items:Switch   PowerHotWaterPumpStatus "Hot Water" (GSwitch,GManagedGoodnight,GManagedSwitch) ["Lighting"] {mios="unit:house,device:303/service/SwitchPower1/Status",mqtt=">[home:house/controller/power_hot_water_pump:state:*:MAP(miosSwitchOut.map)]"}
house.items:Switch   KitchenCoffeeMachineStatus "Coffee" (GSwitch,GManagedGoodnight,GManagedBreakfast,GManagedSwitch) ["Lighting"] {mqtt=">[home:house/controller/kitchen_coffee_machine_status:state:*:MAP(miosSwitchOut.map)]"}
house.items:Switch   KitchenKettle "Kettle" (GSwitch,GManagedGoodnight,GManagedSwitch) ["Lighting"] {mqtt=">[home:house/controller/wemo_spare_status:state:*:MAP(miosSwitchOut.map)]"}
mar

Possible Solution

Determine what's clearing out the JSONDB during service shutdown, and prevent it from clearing the JSONDB entries.

OR;

Determine if we can use the Item name, or a long-enough Hash of it, in the Alexa Service URL instead of allocating/maintaining an "id" (number)

Steps to Reproduce (for Bugs)

  1. Given the 22 Items listed above, or even a small number defined in your system, make a backup copy of the JSONDB file /var/lib/openhab2/jsondb/hue.emulation.lights.json hue.emulation.lights.json-before.txt

  2. Disable the Lighting tag on one of the Items, eg by renaming Lighting to LightingDISABLED. The Item should be somewhere in the middle of the list (the Hue ID's appear unpredictably sequenced)

  3. Observe that the JSONDB file from step 1 has changed, removing the Item from the MAP and leaving a hole (expected)

  4. Shutdown openHAB

    sudo service openhab2 stop

  5. Observe that the JSONDB file from step 1 has been emptied out (unexpected) hue.emulation.lights.json-shutdown.txt

  6. Startup openHAB

    sudo service openhab2 start

  7. Observe that the JSONDB file from step 1 now has completely different value (Hue Id's) for some number of entries. hue.emulation.lights.json-after.txt

In my case, I disabled the Item GManagedFamily by changing it's tag to LightingDISABLED, and the before/after diff of the MAP/JSON looks like:

$ diff *before*  *after*
8c8
<     "value": 21
---
>     "value": 20
18,21d17
<   "GManagedFamily": {
<     "class": "java.lang.Integer",
<     "value": 20
<   },
24c20
<     "value": 22
---
>     "value": 21

ie. the GManagedHolidaySwitch changed from HueID 21 to HueID 20 (previously GManagedFamily` and the other have moved up also.

  1. For one of the Items that's been resequenced, attempt to turn it ON/OFF via Alexa Voice commands (oops)

Context

When the MAP is out of sync with the Amazon Alexa's version of the MAP (from Discover Devices), the wrong device will be operated on by Alexa Voice commands.

eg. "Alexa Turn Kitchen on" might instead turn on the Kettle

Your Environment

Reference

Community discussion: https://community.openhab.org/t/hue-emulation-re-sequences-hueids-on-openhab-service-restart-causes-out-of-sync-with-alexa

@davidgraeff FYI

davidgraeff commented 5 years ago

Determine what's clearing out the JSONDB during service shutdown, and prevent it from clearing the JSONDB entries.

@kaikreuzer Do you have any idea? Am I using the storage service API in a wrong way and is a "keep" flag necessary somewhere?

mrguessed commented 5 years ago

This is the DEBUG log from the shutdown sequence:

2019-01-14 13:00:07.468 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedBreakfast
2019-01-14 13:00:07.481 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedMaster
2019-01-14 13:00:07.493 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item KitchenCoffeeMachineStatus
2019-01-14 13:00:07.511 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedKitchenBlackout
2019-01-14 13:00:07.522 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedKitchenBlinds
2019-01-14 13:00:07.531 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item FamilyHarmony_dvd
2019-01-14 13:00:07.552 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedBlackout
2019-01-14 13:00:07.559 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedKitchen
2019-01-14 13:00:07.584 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedDining
2019-01-14 13:00:07.603 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedFamilyBlackout
2019-01-14 13:00:07.614 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedLiving
2019-01-14 13:00:07.621 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item KitchenIslandLightsStatus
2019-01-14 13:00:07.627 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedBlinds
2019-01-14 13:00:07.640 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item KitchenKettle
2019-01-14 13:00:07.643 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedDinner
2019-01-14 13:00:07.646 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item PowerHotWaterPumpStatus
2019-01-14 13:00:07.648 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item FamilyHarmony_tv
2019-01-14 13:00:07.652 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedGoodnight
2019-01-14 13:00:07.679 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedHolidaySwitch
2019-01-14 13:00:07.693 [DEBUG] [.io.hueemulation.internal.LightItems] - Remove item GManagedMediaLights
2019-01-14 13:00:07.817 [DEBUG] [org.openhab.io.hueemulation         ] - ServiceEvent UNREGISTERING - {org.openhab.io.hueemulation.internal.HueEmulationService}={service.id=457, service.bundleid=260, service.scope=bundle, restrictToTagsWhiteLights=Lighting, component.name=org.openhab.io.hueemulation.internal.HueEmulationService, service.config.label=Hue Emulation, component.id=298, createNewUserOnEveryEndpoint=false, restrictToTagsSwitches=Switchable, restrictToTagsColorLights=ColorLighting, discoveryHttpPort=80, pairingTimeout=600, pairingEnabled=false, service.config.category=io, service.config.description.uri=io:hueemulation, service.pid=org.openhab.hueemulation} - org.openhab.io.hueemulation
2019-01-14 13:00:07.841 [DEBUG] [org.openhab.io.hueemulation         ] - ServiceEvent UNREGISTERING - {javax.servlet.ServletContext}={osgi.web.version=2.4.0, osgi.web.contextpath=/, service.id=458, osgi.web.symbolicname=org.openhab.io.hueemulation, service.bundleid=260, service.scope=singleton, osgi.web.contextname=default} - org.openhab.io.hueemulation
2019-01-14 13:00:07.878 [DEBUG] [org.openhab.io.hueemulation         ] - BundleEvent STOPPING - org.openhab.io.hueemulation
2019-01-14 13:00:07.879 [DEBUG] [org.openhab.io.hueemulation         ] - ServiceEvent UNREGISTERING - {org.osgi.service.cm.ManagedService}={service.id=456, service.bundleid=260, service.scope=singleton} - org.openhab.io.hueemulation
2019-01-14 13:00:07.880 [DEBUG] [org.openhab.io.hueemulation         ] - BundleEvent STOPPED - org.openhab.io.hueemulation

Those "Remove Item..." log entries, from removed() in (LightItems.java), appear to be the ones that are clearing out the Storage Service that's used by the Hue Emulation.

thekev commented 5 years ago

I'm thinking it should call close() before shutdown gets a chance to remove all the items. I'm looking into how this might be accomplished.

davidgraeff commented 5 years ago

Had anybody any success in this matter? I mean it is quite serious somehow. I'm myself unfortunately busy with other areas in OH

bademux commented 5 years ago

How about using Openhab's item name as Hue's uniqueId? As far as I know item name is immutable and unique, and according to HueHab documentation it is string. Alexa work pretty well with such ids

davidgraeff commented 5 years ago

The original hue bridge uses numbers only that's why we have numbers as well. It is actually even better to save the hue number as item meta data and not separate from the item.

bademux commented 5 years ago

I didn't know that item object can store metadata within itself. But in that case we still have an issue - sequence should be handled. I wonder if item name hashcode hashcode is distributed well enough. After all the simplest and fastest solution is item name.

davidgraeff commented 5 years ago

I do not have a Hue developers account. If anyone can verify if arbitrary names can be used and quote that part that'd be helpful. If they restrict it to numbers only though we still need the mapping.

bademux commented 5 years ago

My openhab plugin (PoC) works with Alexa using item names.

There no description for root element, but in example we have string ("1")

{
"1": {
        "state": {
....
        },
....
}

https://developers.meethue.com/develop/hue-api/lights-api/#get-all-lights

A bit of internals (can be threaded as argument to use associated numeric ids): "Yes, as Wim says it is not possible to change a light ID. What you mention will work though, as deleting a light and finding it again will result in it being assigned the next available light id, but this is of course a bit messy and strictly speaking not a change light ID. Hue Support - Steve" https://developers.meethue.com/forum/t/change-light-id/4167

davidgraeff commented 5 years ago

Yes I know that it's not an array, but an associate map with stringed keys. But I guess some sloppy apps and hue API consumers out there are converting those keys to numbers, because the original Hue bridge will only ever have numbers. That's my concern.

I mean ideally the hue emulation also works with the original Hue app and all kind of 3rd party apps. And that's why I want the API doc to clearly say "all imaginable keys can be used, but we will use numbers for now" or so.

bademux commented 5 years ago

@davidgraeff yes I totally agree with you. The answer, depends on purpose of the Openhab Addon.

In my opinion HueEmulator should be good enough to support several popular devices\apps. 100% emulation is not necessary and unreal :(

But back to your proposition to attach hueId to metadata "It is actually even better to save the hue number as item meta data and not separate from the item." How you will achieve that, storing metainfo in tags?

davidgraeff commented 5 years ago

How you will achieve that, storing metainfo in tags?

No. That's what you would have done in 2.3, I guess. But since OH 2.4 there is a meta-data system in place. This is already used by haBot and by the auto-update feature. It's like attaching configuration to an Item, but namespaced. And in this case we would claim the namespace "hueemulation".

bademux commented 5 years ago

Well, then it sounds like almost perfect solution. Could you please share the link to the doc or sources?

davidgraeff commented 5 years ago

https://www.eclipse.org/smarthome/documentation/concepts/items.html#item-metadata

thekev commented 5 years ago

Okay, I think I see a way here. If the item metadata stores the hueemulation ID, and the addon only stores the next-id integer, then we can use a simple incrementing value, which will only be assigned to a candidate item if missing from its metadata.

The max value of this ID is...? The hue bridge is said to support 50, with some reports of more being accomplished. Not clear what the protocol spec says. If too small, then need to get more tricksy with doing things like zwave: increase until max, then reuse IDs previously vacated. I'm thinking a safe starting point for max is 255.

davidgraeff commented 5 years ago

Yup. 255 sounds reasonable. From that point on we will still expose further items but add a big warning to the log file.

bademux commented 5 years ago

sorry being like a broken radio, but MAX_INT as safe as 255. Documentation refers to 50 devices, not the length or type of hueId. Imho, just go with MAX_INT or better with timestamp (no need to keep max value). If there will be problem it will be reported.

davidgraeff commented 5 years ago

I think the old maximum came from the fact that the first hue bridge was an embedded system with a reserved memory area for storing lights. The second edition is now a linux based system with AFAIK 256 mb ram, more than enough to store hundreds of lights.

And zigbee allows 65536 devices in one single mesh.

schniddsel commented 5 years ago

I'm thinking it should call close() before shutdown gets a chance to remove all the items. I'm looking into how this might be accomplished.

Did you check the "close()" option? I think if this works you could do a hotfix and invest some more time in the meta solution afterwards. I by myself ran into the problem yesterday after rebooting my rasp after some updates. "Alexa Kitchen off" turned off my tv... well. Sad girlfriend watching Netflix.

Is it possible to overwrite the jsondb? So that i could back it up on my pc and overwrite it after a reboot of openhab.

I really appreciate your work guys. It is great to have such a feature without having to use the... let's say quite instable cloud-service.

davidgraeff commented 5 years ago

@schniddsel There is no working solution yet, as no jsondb maintainer has commented so far. Let's try again:

@openhab/core-maintainers

maggu2810 commented 5 years ago

I read the first comment that describes the problem twice but I still don't understand it. That's surely caused by the fact that I never used the alexa integration and hue emulation of openHAB.

@davidgraeff Can you give it a try to explain me the situation? Or perhaps another maintainer that used that stuff already could comment.

The person for the jsondb storage internals could be @cdjackson as he has written that implementation. But I assume it is not storage implementation dependent (but cannot judge without understanding the problem at all).

davidgraeff commented 5 years ago

tl;dr

The hue emulation requires a storage. Before I refactored it, it used custom files, but those were not part of the OH backup system, so I switched to the json storage API.

And now the problem: The json storage resets its content after each start (not on shutdown though). And I really do not understand why, the API doc does not say that it auto-prunes itself or similar.

maggu2810 commented 5 years ago

The json storage resets its content

What does it mean? Reset to which state? Does the storage service provides an empty storage after each restart?

davidgraeff commented 5 years ago

Does the storage service provides an empty storage after each restart?

Yes exactly. And the underlying file is cleared.

If the service is restarted during runtime, it picks up all the stored information correctly.

maggu2810 commented 5 years ago

Did you check the file content as long as openHAB is not running? So, are the wrong information in the file or is it not deserialized correctly?

maggu2810 commented 5 years ago

Yes exactly. And the underlying file is cleared.

So, the file exists and it is empty (size = 0 bytes)?

davidgraeff commented 5 years ago

So, the file exists and it is empty (size = 0 bytes)?

I need to double check, but afaik yes it's an empty file. Maybe I'm just using the service in the wrong way.

maggu2810 commented 5 years ago

You could modify the put method to get some more information.

https://github.com/openhab/openhab-core/blob/a37ccea/bundles/org.openhab.core.storage.json/src/main/java/org/eclipse/smarthome/storage/json/internal/JsonStorage.java#L134

E.g.

public @Nullable T put(String key, @Nullable T value) {
    if (logger.isTraceEnabled()) {
        logger.trace("put to storage: file: {}, key: {}, value: {}", file, key, value, new RuntimeException("trace stacktrace"));
    }
    ...
davidgraeff commented 5 years ago

I was afraid that core bundle debugging is required. I have not fired up an openhab-core development setup yet. The IDE situation is not stable enough, especially if you put core and addon bundles into the same workspace.

maggu2810 commented 5 years ago

You do not need "core bundle debugging".

You just need to modify that bundle (e.g. using a simple text editor), compile on command line and use that JAR it in your launch configuration.

You could also build the modified bundle add use that in your Karaf based OH runtime.

Then you only need to set the log level and read the log after a start-shutdown-cycle.

mdeneen commented 5 years ago

So that's why my Alexa devices were horribly confused every time I restarted the service. This is further compounded by Amazon caching the results on their servers, which resulted in me having four or five duplicates and no way to tell which was the right one.

Is there anything which I can do to help resolve this?

davidgraeff commented 5 years ago

I'll migrate the service to the new build system tomorrow. And at that point I'll swap our made up hue IDs with openHAB item names.

I will test that with my Echos here. If that doesn't work I'll use item meta data to store assigned IDs.

Either way, after that change is merged and snapshots are working again, the emulated hue devices have to be discovered once more :/ (aka breaking change)

davidgraeff commented 5 years ago

Done in #5243