Ikea Tradfri bulbs support

[cswrd]

This is a feature request to have a better support for Ikea Tradfri bulbs.

In comparison to v0.7.1 the compatibility of v0.8.0 with these bulbs got reduced. Here is why: as far as I know and based on some API tests I made to confirm that with these bulbs (LED2101G4) and the latest Firmware (1.0.003) at least these properties are different to hue bulbs:

1. ct and bri sent within 1 API request will apply only ct. Unfortunately, fetching the data (see below) will respond with the intended values for some seconds, even though bri won't be applied.
2. ct and bri have to be sent with 2 individual requests and with a little delay in-between. Otherwise the ongoing change, e.g. brightness at 254 set to 1 (which is not instantaneous) is aborted in between. A delay of 100ms seems reasonable. Or send with a transitiontime of 0 (source).

As mentioned in #2 with v0.7.1 a workaround is possible: just run at least 2 hue schedulers, one responsible for brightness and another responsible for color temperature and make sure to schedule both properties at different times. Only the retry behaviour might violate number 2 but this is very unlikely.

With v0.8.0 this workaround yields to hue schedulers sending requests concurrently when a light is turned on due to the fast event mechanism. And the change from one scheduler would be interpreted by the other scheduler as a manual change, probably.

For sure another workaround would be to schedule either of these properties and never schedule the other property.

This is the response from such a Tradfri bulb:

https://ip/api-key/lights/7

```js { "state":{ "on":true, "bri":254, "ct":250, "alert":"select", "colormode":"ct", "mode":"homeautomation", "reachable":true }, "swupdate":{ "state":"notupdatable", "lastinstall":"2023-07-06T17:51:53" }, "type":"Color temperature light", "name":"Wohnzimmer Vorn #1", "modelid":"TRADFRI bulb E14 WS globe 470lm", "manufacturername":"IKEA of Sweden", "productname":"Color temperature light", "capabilities":{ "certified":false, "control":{ "ct":{ "min":250, "max":454 } }, "streaming":{ "renderer":false, "proxy":false } }, "config":{ "archetype":"classicbulb", "function":"functional", "direction":"omnidirectional" }, "uniqueid":"94:34:69:ff:fe:bc:25:c1-01", "swversion":"1.1.003" } ```

edit: Actually, hue scheduler recognizes a manual change for them with each schedule. I am not sure why, yet. When I check the lights with the above API request the intended brightness is set correctly.

[stefanvictora]

Thanks for your investigation. I unfortunately don't have any experience with Ikea Tradfri bulbs and wasn't aware of any limitations regarding API calls directed at those bulbs.

As far as I understood there are multiple issues at the moment, please correct me if I forgot something:

1. Ikea Tradfri bulbs can't handle multiple property changes in a single API call
2. If split into multiple calls, there needs to be a short delay between them
3. Multiple instances of the scheduler cause too many request rejections by the Hue bridge
4. The manual change tracking of the Hue Scheduler prevents some updates if multiple instances are running in parallel

For the second point, there could be a potential workaround that could restore compatibility with v0.8.0, when running multiple instances. In v0.8.0 the --power-on-reschedule-delay global configuration flag was added (initially forgot to mention it in the release notes), which controls the delay in ms after a light-on event was detected and when the scheduler should reapply the desired light states. The default value is 150ms, but you can adjust this value for the multiple instances, which should prevent the scheduler to fire pending updates all at the same time, when the lights are turned on.

Regarding the third point: While Hue Scheduler has an internal rate limiter that should prevent too many requests in parallel, it can't coordinate those limits across multiple instances running in parallel. You could however lower the rate limit for each instance via the --max-requests-per-second global configuration flag. The default is set to 10 requests per second. If you come across this error again, I would be interested in the exact error message the bridge is throwing, as I could probably improve such error handling in the future.

Regarding the fourth point: Thank you for bringing up the manual change tracking. Looking into it I found a bug for tracking changes of states where only the brightness is set. Internally, the scheduler always compared the current color mode of the light with the color mode of the last applied state. If they differed, a manual override would be detected. For states that only define the brightness, no expected color mode can be derived, but the scheduler failed to ignore the color mode in such cases and therefore incorrectly detected a change. I published a fix for this with version v0.8.0.1. Let me know, if this solves this issue.

To fully solve the issue with Ikea Tradfri bulbs, it would be interesting to know if this limit of "one property per request" applies to all properties i.e. ct, bri, hue, sat, xy or just ct and bri. If we know what the bulbs expect, I could adapt the API calls for those specific models explicitly, so that you wouldn't need to run multiple instances to begin with.

[cswrd]

Thank you for this quick hotfix, I can confirm it solved that brightness-only bug, even for Tradfri bulbs. That also means that basic support for Tradfri bulbs with only 1 property being scheduled is possible now again.

And I've got very good news. Trafdri support is almost there =). I made a couple more observations and probably found a little bug. If that's fixed, Tradfris work well enough.

Actually, transitiontime is supported by those bulbs (I spelled it wrong in my first post and my tests, initially). It works fine as long as only 1 state is being changed. And it's even possible to apply at least 2 states with 1 API request. But... one needs to send it explicitly with a transitiontime of 0. That's where tr and tr-before come very handy. I found that information in this post and can confirm it works with my powershell tests.

It sounds like it should already work with hue scheduler. But after testing this approach with it and enabling the TRACE log I've noticed that the tranisitontime is sent with the value 3 instead of 0. Resulting in those unfinished state changes and hue scheduler detecting manual changes, hence aborting further schedules. Interestingly, the state change on application start is sent with a transitionTime of 0 and that very first state change works as intended.

So if that'd be fixed sufficient Tradfri support would be available. One could add a note like this into the readme: "Most Tradfri bulbs suffer a firmware bug when sending 2 or more property changes (e.g. bri and ct) concurrently to the bulb with a transitiontime above 0. It won't apply both values. This also causes an unintentinal manual change detection in hue-scheduler that stops further schedules. So if you want to schedule more than 1 state at the same time make sure to set tr and tr-before to 0 for these bulbs, explicitly. If you still want to have some kind of smooth transition with multiple properties, you could schedule a bunch of minor state changes on your own to imitate the transition behaviour.

For me that'd be sufficient enough and would save the time for implementing workarounds directly into hue scheduler that might even become obsolete at some point (If Ikea would fix the firmware). But it's up to you, just let me know.

[stefanvictora]

This is great to hear, thank you for the quick test and your further investigations into the issue. Interesting to see how this has been solved by other projects by setting the transition time to 0 -- thanks for the link!

I'm also wondering why the transition time is set to a value of 3, instead of 0, if it has been explicitly set. Can you maybe share the relevant state configurations? One though that comes to mind is that tr-before should probably not be set at all, if the goal is to have an exact transition time of 0 in the API call, as tr-before is right now only intended for dynamically calculating transition times. Maybe the scheduler still detects some time difference of a few ms even if tr-before is set to 0, which could then in turn be used for the transition time instead of the explicit tr:0. So better just set tr:0 and don't set tr-before at all, as it's not needed and intended for this use case.

And thank you for the suggested warning note, I agree that it makes sense to place this in the Readme so others are aware of this issue.

I will still think if there is maybe a clean way to support this natively, but you are right, it might become at some point obsolete.

[cswrd]

Jippie, not setting tr-before does the trick to send the transitiontime as 0. I confirmed the behvaiour with this setting:

Wohnzimmer Vorn  19:04  bri:1    ct:2200  tr:0
Wohnzimmer Vorn  19:05  bri:254  ct:4000  tr:0
Wohnzimmer Vorn  19:06  bri:1    ct:2200  tr:0
Wohnzimmer Vorn  19:07  bri:254  ct:4000  tr:0

Adding tr-before:0 or tr-before:0s will send a 5 today instead of a 3 like yesterday. I didn't expect that to be the solution, as I wanted to be as specific as possible. Thought it's a fall back to some default value or so:

Wohnzimmer Vorn  19:08  bri:1    ct:2200  tr:0  tr-before:0
Wohnzimmer Vorn  19:09  bri:254  ct:4000  tr:0  tr-before:0
Wohnzimmer Vorn  19:10  bri:1    ct:2200  tr:0  tr-before:0
Wohnzimmer Vorn  19:11  bri:254  ct:4000  tr:0  tr-before:0

off-topic: So at that point thanks to you I can achieve everything I ever wanted with my smart lights at home. I can also run just 1 hue-scheduler now. Both a further increased support for Tradfri bulbs and the "Continuous transition management" would probably "only" reduce the huge size of configuration for me but not change the behaviour from a user perspectve. Sure, the ability to achieve the same with way less configuration workarounds would still be very welcome.

fyi: The most impact for my day to day usage would be those things we already talked about but are limited by the Hue API: ideally an instant reaction time after a dumb wall switch turn on as well as the fast turn on/off recognition to clear manual changes immediately.

I'll let you know when I spot further issues. Thank you very much for that great peace of software.

[cswrd]

I just built a little excel helper to create the schedule interpolation entries. Now I have about 800 schedule entries. The app requires >15 minutes to apply the schedules now. I am curious if it'll work stable.

[stefanvictora]

Hi @cswrd, thank you for bringing the issue of long startup times to my attention. I investigated the potential causes and managed to make some significant improvements, which are now available in the newly released v0.10.0. Feel free to check it out and let me know if it fixes the issue!

To summarize: I added a cache for parsing the solar and local times, as these computations are performed frequently when Hue Scheduler tries to sort the states and calculate their respective end times during startup.

[cswrd]

Hi @stefanvictora I just migrated from v0.9.0 to v0.11.0 some minutes ago. I'll let you know if I find some issues.

Meanwhile, I don't use 800 entries for my Tradfris anymore. Not due to long startup times. But there were a lot of unintended manual override detections. I managed to find some issues in my schedules that were triggered due to the changed sun times over the year. Though, some of the manual overwrite detections still didn't make any sense. I double checked the settings and console output multiple times. With a shorter interval (e.g. 5 minutes in between the cycles instead of every minute) it's working quite well. I guess this might be related to some delays and race conditions in the state detection limited by the hue API. My second guess is the color comparison that was not using thresholds up until v0.11.0. What do you think, should schedules and the overwrite detection work even when I schedules on a per-minute-basis?

[stefanvictora]

Hi @cswrd, thanks for the updates. This is a difficult question to answer since we don't fully understand what the bridge is doing internally. It's possible that the detected changes are due to race conditions between the explicit updates and the periodic polling performed by the bridge. However, you might also be correct that the color comparison is causing the issue. Not all bulbs support every color, so a conversion is done internally to fit their supported range (color gamut). Normally, Hue Scheduler uses the color gamut each light provides to convert colors beforehand, but it's challenging to get this 100% right. Additionally, some third-party bulbs might not report their supported gamut correctly, or at all.

The new comparison using thresholds should hopefully minimize such issues. However, please note that v0.11.0 does not yet use thresholds for color temperature (CT) comparisons. This will be addressed in the upcoming v0.12.0 release, which I am currently working on. Another known issue with v0.11.0 that I recently discovered is that, due to the increased performance of the scheduler, the bridge might now throw "Rate Limit Exceeded" errors even for simple status lookups. Hue Scheduler currently struggles with this unexpected bridge behavior, which, in the worst case, can cause it to lose track of some states. This issue will also be resolved in the next version.

I will keep you updated as soon as the new version is released. If you encounter those issues with v0.11.0, you can try out a pre-release version with the fixes by using the "dev" tag of the Docker image. However, please note that I can't guarantee its stability yet.

And to also answer your question: In theory, both the bridge and Hue Scheduler should support a per-minute update of lights. Unfortunately, there is currently no debug log available to investigate the details of what exactly caused the override to be detected.

[cswrd]

@stefanvictora After further debugging my config and the logs I realized I had a misconception about the time constants shifting throughout the year. I thought It'd be sufficient to check if my configuration would work on the 21st of june and the 21st of december. I did this by injecting a different datetime to the container. Just like testing "extreme scenarios".

For example I used 2 configurations: sunset-50 and golden_hour. They differ by 20 - 30 minutes at both mentioned dates on my location. However, during the fall, the golden_hour might be earlier as sunset-50. This triggered these "ouf of order" / unintentional schedules and when they result in a schedule faster than ~3 minutes before the last one, this seems to trigger the manual detection which stops the automation.

So still a layer 8 problem. Would a GUI help on this? How do you verify your schedules would "work" on every day in the year? (edit: there is a similar discussion in #9)