Home Assistant MQTT format now requires color_mode to be sent

[VossenTech]

Describe the bug

Updates to home assistant mqtt light json format require an additional parameter to be sent in the json. Parameter: "color_mode" Possible values: "onoff", "brightness", "color_temp", "hs", "xy", "rgb", "rgbw", "rgbww", "white"

Without sending this parameter the entities no longer have brightness, color_temp, or RGB attributes due to the color_mode being unknown. The lights are still fully functional using manual control however any automations that utilize any of these attributes as triggers or checks the current values are broken.

I did some quick testing and it seems to me that when milight hub is sending "bulb_mode":"color" HA wants it to also be sending "color_mode":"rgb", and when its "bulb_mode":"white" than HA wants "color_mode":"color_temp". Hopefully that makes it an easy thing to add.

[AlexKusnezov]

@VossenTech my workaround ist tu use optimistic, that way brightness, color and temp will show in home assistant, here's my config:

- schema: json
  name: "Deckenleuchte (Studio)"
  unique_id: deckenleuchte_studio
  command_topic: "milight/command/0x1F0F/rgb_cct/1"
  state_topic: "milight/state/0x1F0F/rgb_cct/1"
  qos: 1
  optimistic: true
  retain: true
  color_temp: true
  rgb: true
  brightness: true
  effect: true
  effect_list:
    - night_mode
    - white_mode
    - '0'
    - '1'
    - '2'
    - '3'
    - '4'
    - '5'
    - '6'
    - '7'
    - '8'
  device:
    configuration_url: "http://192.168.2.102/"
    connections: [["ESPMH", "0x1F0F"]]
    identifiers: "0x1F0F"
    manufacturer: "Lighteu"
    model: "LT-NGU10-01 (FUT103)"
    name: "Deckenleuchte (Studio)"

works perfect:

[VossenTech]

Just tried optimistic mode and it did fix the brightness and rgb_color issue, however it seems to set the color_mode to hs (presumably HA's default) so color_temp is still not a state attribute for the light. It does work form the UI same as the screenshots you posted but if you go to develper tools > states and look at the attributes you will see that color_temp is missing so it cannot be referenced in automations and scripts. Since hs is an RGB mode it doesnt have any value other than 0.0 when the light is in white mode so there is no value that will give you the current light temp.

[AlexKusnezov]

@VossenTech this is how it looks on my end: White Setting:

Color Setting:

seems right to me, or does color_mode have to be rgb when setting color?

[VossenTech]

I stand corrected. Just turning the light off and on while in white mode didn't change color_mode to color_temp, however after manually changing the temp from the HA UI it switched from hs to color_temp and stayed that way though off/on cycles so that seems to be a good workaround for me for now. Thank you for suggesting it.

I'm not really concerned with hs vs color since they both provide the rgb_color attribute but if someone else is looking for that specifically it can be set using the color_mode and supported_color_modes options in the config along with the optimistic mode.

  - name: "Office Lights"
    command_topic: "milight/0x20/rgb_cct/1"
    state_topic: "milight/states/0x20/rgb_cct/1"
    schema: json
    color_mode: true
    supported_color_modes: ["rgb", "color_temp"]
    optimistic: true
    brightness: true
    effect: true
    effect_list:
        - white_mode
        - night_mode
        - 0
        - 1
        - 2
        - 3
        - 4
        - 5
        - 6
        - 7
        - 8

[sidoh]

Sorry for the terribly late reply. I've been extremely lazy about upgrading HomeAssistant and just got around to it now.

It seems the MQTT discovery stuff isn't quite right (will publish a fix for that soon), but I believe the devices are working fine after that:

This is with HomeAssistant 2023.9.0. If you guys can help me replicate the issue you're seeing, happy to take a crack at fixing it.

[VossenTech]

Same issue still exists with 2023.9.0, everything shows up correctly in the UI and is functional via manual control but because no value for 'color_mode' is provided there are no attributes on the light for RGB, color temp, or brightness.

These are the attributes available:

As a result any automations that rely on checking the current state of the light are broken. Enabling optimistic mode brings back the attributes once you have manually changed the white temp or RGB color of the light once.

I still believe the solution is to add 'color_mode' to the mqtt messages since an HA dev told me it was now a required field. Here a link to issue I had opened with HA before opening one here: https://github.com/home-assistant/core/issues/92224

[sidoh]

I'm just using MQTT discovery, which does not use optimistic mode. I believe I see all of the expected fields here (tell me if I'm missing something):

It probably is the case that some fields are not filled out in the state message until you make changes to the relevant mode (for example, the state of brightness in RGB mode is not known if the hub has never controlled or seen packets that control brightness in RGB mode). So it might be the case that everything works out when you make state changes regardless of optimistic mode or not.

[VossenTech]

It looks like that light is in white mode but doesn't have a color_temp or any other attribute that will tell you what temperature the light is. I would expect either color_temp, color_temp_kelvin, or really any value that could be used to determine the white temp.

I believe when I first started troubleshooting that is what it looked like for me, I think updating to the newer MQTT light spec (using supported_color_modes: ["rgb", "color_temp"] instead of rgb:true and color_temp:true) changed it to what I see now. But without optimistic mode both cases are missing a temp attribute.

[sidoh]

Aaah okay, understood. Let me see if adding that fixes things. Thanks for your patience -- I don't use anything that would care about these attributes so was hard for me to see what the issue was.

[sidoh]

Does this look right?

[VossenTech]

The first screenshot doesn't seem to have any white temp value, the second appears to be color and the rgb_color value looks good, along with hs. I think the logic from my first post should would if you try adding color_mode to the json payload.

it seems to me that when milight hub is sending "bulb_mode":"color" HA wants it to also be sending "color_mode":"rgb", and when its "bulb_mode":"white" than HA wants "color_mode":"color_temp". Hopefully that makes it an easy thing to add.

[sidoh]

Yeah, that's what's happening in these screenshots. ~There is a white temp value, I think?~ nevermind I'm dumb

When in white mode, this is the JSON payload:

{"state":"ON","brightness":255,"color_temp":370,"bulb_mode":"white","color":{"r":255,"g":255,"b":255},"color_mode":"color_temp"}

when in color mode, it's:

 {"state":"ON","brightness":255,"bulb_mode":"color","color":{"r":12,"g":69,"b":255},"color_mode":"rgb"}

I think this is what we wanted, right?

Just want to confirm I'm doing the right thing before investigating further (any help appreciated, of course)

[sidoh]

The change is included in this beta release if you want to try it out yourself: https://github.com/sidoh/esp8266_milight_hub/releases/tag/1.10.10-beta1

[VossenTech]

Those payloads look good to me, I updated to that beta and enabled color_mode in the group state fields. I don't seem to get color_mode in the payload if it is in white mode but it is in the payload for color mode and that seems to update HA correctly.

white_mode payload: {"state":"ON","brightness":189,"color_temp":233,"bulb_mode":"white","color":{"r":255,"g":255,"b":255},"effect":"white_mode","device_id":32,"group_id":1,"device_type":"rgb_cct"}

color_mode payload: {"state":"ON","brightness":145,"bulb_mode":"color","color":{"r":12,"g":16,"b":255},"device_id":32,"group_id":1,"device_type":"rgb_cct","color_mode":"rgb"}

[sidoh]

Hmmm I think I would only expect this to be happening if the bulb state is unknown. Can you try adjusting the color temperature when it's in that state?

[sidoh]

Pushed this passing integration test for reference:

https://github.com/sidoh/esp8266_milight_hub/commit/076c292ea28da76e4c0643055135a63221c9beae

[VossenTech]

Looks like its something to do with including device_id and group_id. If I enable either of them independently then I get "color_mode":null when in white mode. If I enable both then color_mode is missing entirely in white mode. If I disable both it seems to be working as expected. Color mode works correctly regardless of these settings.

[VossenTech]

On a separate note it would be nice if the brightness stayed the same level when switching between white and color modes. They seem to have separate saved brightness'. I can open a separate issue for that if you would like.

[sidoh]

Looks like its something to do with including device_id and group_id. If I enable either of them independently then I get "color_mode":null when in white mode. If I enable both then color_mode is missing entirely in white mode. If I disable both it seems to be working as expected. Color mode works correctly regardless of these settings.

Hmm that's strange. Will try to reproduce.

On a separate note it would be nice if the brightness stayed the same level when switching between white and color modes. They seem to have separate saved brightness'. I can open a separate issue for that if you would like.

This is just how the milight bulbs behave -- the state is just being represented as-is. You should be able to control the brightness in any automations you're setting up. So for example, change the bulb to color mode from color temp mode and set the brightness to what you want. If you want to copy it over from the other mode I think you can probably do that?

It's probably possible to add a setting to keep them in sync but would like to avoid that unless it's a pretty big pain to do it outside of the hub.

[VossenTech]

This is just how the milight bulbs behave -- the state is just being represented as-is. You should be able to control the brightness in any automations you're setting up. So for example, change the bulb to color mode from color temp mode and set the brightness to what you want. If you want to copy it over from the other mode I think you can probably do that?

It's probably possible to add a setting to keep them in sync but would like to avoid that unless it's a pretty big pain to do it outside of the hub.

Yeah, controlling it with automations isn't a problem, this one is more of a manual control annoyance that has been exacerbated by HA's choice to put brightness, color, and white temp on separate tabs for light control. Having to turn on the light then go to the color tab then back to the brightness tab feels like a lot of unnecessary clicks (especially when your kids want the colors changed many times per day). I'm sure I can just write an automation that sets the brightness when it switches from one to the other and solve it myself though.

[sidoh]

Sorry this took so long. Got way deep down a rabbit hole with something else.

I'm not able to reproduce, unfortunately. Even setting device_id/group_id fields, I still get expected behavior.

I'm just using the integration test I linked here, which does the following:

1. Sets these group state fields:

     @client.patch_settings({
       group_state_fields: [
         "status",
         "level",
         "kelvin",
         "hue",
         "saturation",
         "color_mode",
         "hex_color",
         "device_id",
         "group_id"
       ]

2. Patches state for an otherwise unknown device to status="ON", color: '#ff0000'. It then expects color_mode to be rgb (which it is)

If you can help me reproduce it'd help a bunch!

[VossenTech]

Try adding effect with device_id and group_id, it seems to be something about that combination.

[sidoh]

Yeah, the example I pasted above is using both device_id and group_id. I'm not seeing the same behavior.

Can you share the output of /settings? (make sure to sensor any passwords that might be present). Might make it easier to reproduce.

[VossenTech]

Right, I’m saying add ‘effect’ to that list in addition to device_id and group_id. I can share my settings when I’m in front of my computer again.

[sidoh]

Haha, sorry. Failing my reading comprehension today.

I'm getting the same thing with effect, device_id, group_id all in there together (i.e., am not able to reproduce).

Hopefully will be able to to with your full settings.

[VossenTech]

{ "admin_username": "redacted", "admin_password": "redacted", "ce_pin": 4, "csn_pin": 15, "reset_pin": 0, "led_pin": -2, "radio_interface_type": "nRF24", "packet_repeats": 50, "http_repeat_factor": 1, "auto_restart_period": 0, "mqtt_server": "redacted:1883", "mqtt_username": "redacted", "mqtt_password": "redacted", "mqtt_topic_pattern": "milight/:device_id/:device_type/:group_id", "mqtt_update_topic_pattern": "", "mqtt_state_topic_pattern": "milight/states/:device_id/:device_type/:group_id", "mqtt_client_status_topic": "", "simple_mqtt_client_status": false, "discovery_port": 48899, "listen_repeats": 0, "state_flush_interval": 10000, "mqtt_state_rate_limit": 500, "mqtt_debounce_delay": 500, "mqtt_retain": true, "packet_repeat_throttle_sensitivity": 0, "packet_repeat_throttle_threshold": 200, "packet_repeat_minimum": 3, "enable_automatic_mode_switching": false, "led_mode_wifi_config": "Fast toggle", "led_mode_wifi_failed": "On", "led_mode_operating": "Off", "led_mode_packet": "Flicker", "led_mode_packet_count": 3, "hostname": "milight-hub-south", "rf24_power_level": "MAX", "rf24_listen_channel": "LOW", "wifi_static_ip": "", "wifi_static_ip_gateway": "", "wifi_static_ip_netmask": "", "packet_repeats_per_loop": 10, "home_assistant_discovery_prefix": "", "wifi_mode": "n", "default_transition_period": 500, "rf24_channels": [ "LOW", "MID", "HIGH" ], "device_ids": [32], "gateway_configs": [], "group_state_fields": [ "state", "brightness", "color_temp", "bulb_mode", "computed_color", "effect", "device_id", "group_id", "color_mode" ], "group_id_aliases": { "Office Lights": [ "rgb_cct", 32, 1] } }

[sidoh]

Looks like I'm still not seeing it, even with exactly the same settings. Can you try repeating what I'm doing directly to see if it reoccurs?

For the sake of completeness, here are the settings I'm trying with (only difference is I removed the username/password):

{"admin_username":"","admin_password":"","ce_pin":4,"csn_pin":15,"reset_pin":0,"led_pin":-2,"radio_interface_type":"nRF24","packet_repeats":50,"http_repeat_factor":1,"auto_restart_period":0,"mqtt_server":"redacted:1883","mqtt_username":"redacted","mqtt_password":"redacted","mqtt_topic_pattern":"milight/:device_id/:device_type/:group_id","mqtt_update_topic_pattern":"","mqtt_state_topic_pattern":"milight/states/:device_id/:device_type/:group_id","mqtt_client_status_topic":"","simple_mqtt_client_status":false,"discovery_port":48899,"listen_repeats":0,"state_flush_interval":10000,"mqtt_state_rate_limit":500,"mqtt_debounce_delay":500,"mqtt_retain":true,"packet_repeat_throttle_sensitivity":0,"packet_repeat_throttle_threshold":200,"packet_repeat_minimum":3,"enable_automatic_mode_switching":false,"led_mode_wifi_config":"Fast toggle","led_mode_wifi_failed":"On","led_mode_operating":"Off","led_mode_packet":"Flicker","led_mode_packet_count":3,"hostname":"milight-hub-south","rf24_power_level":"MAX","rf24_listen_channel":"LOW","wifi_static_ip":"","wifi_static_ip_gateway":"","wifi_static_ip_netmask":"","packet_repeats_per_loop":10,"home_assistant_discovery_prefix":"","wifi_mode":"n","default_transition_period":500,"rf24_channels":["LOW","MID","HIGH"],"device_ids":[32],"gateway_configs":[],"group_state_fields":["state","brightness","color_temp","bulb_mode","computed_color","effect","device_id","group_id","color_mode"],"group_id_aliases":{"Office Lights":["rgb_cct",32,1]}}

Here are the commands I'm running to test with the associated responses:

$ curl -X DELETE '10.133.8.59/gateways/32/rgb_cct/1?blockOnQueue=true'
{"success":true}%                                                                                                                                                                                               

$ curl -X PUT -H 'content-type: application/json' -d '{"status":"ON","kelvin":0}' '10.133.8.59/gateways/32/rgb_cct/2?blockOnQueue=true'
{"state":"ON","color_temp":153,"bulb_mode":"white","color":{"r":255,"g":255,"b":255},"effect":"white_mode","device_id":32,"group_id":2,"color_mode":"color_temp"}% 

$ curl -X PUT -H 'content-type: application/json' -d '{"status":"ON","hue":0}' '10.133.8.59/gateways/32/rgb_cct/2?blockOnQueue=true'
{"state":"ON","bulb_mode":"color","color":{"r":255,"g":0,"b":0},"device_id":32,"group_id":2,"color_mode":"rgb"}% 

In each case it looks like I'm seeing what I'd expect (e.g., color_mode is color_temp in the first command and is rgb in the second).

alternatively f you can hit me with some steps to reproduce the problem that works too!

[VossenTech]

I did some testing and it looks like you are returning different payloads to the api and mqtt.

I did the delete command and then ran the following to turn on the light: curl -X PUT -H 'content-type: application/json' -d '{"state":"ON","brightness":200,"color_temp":201}' '<IP Address>/gateways/32/rgb_cct/1?blockOnQueue=true'

I changed it from your original example to match the command home assistant is sending via MQTT.

The response payload from the API was: {"state":"ON","brightness":199,"color_temp":201,"bulb_mode":"white","color":{"r":255,"g":255,"b":255},"effect":"white_mode","device_id":32,"group_id":1,"color_mode":"color_temp"}

However the response published to the MQTT state topic was: {"state":"ON","brightness":199,"color_temp":201,"bulb_mode":"white","color":{"r":255,"g":255,"b":255},"effect":"white_mode","device_id":32,"group_id":1,"color_mode":null}

I tested with some other commands and got varying results, it seems like once a command came in via MQTT color_mode would stop working in white mode (in mqtt only) regardless of what I sent via api call but the above example was able to reproduce the error every time for me, hopefully it will for you as well.

[sidoh]

That's very helpful!

Figured it out. The JSON buffer for MQTT state messages was a fixed 200 bytes and was overflowing with all of the fields enabled. Going to bump it up to 1 KB and we should be good.

[sidoh]

Give this release a shot:

https://github.com/sidoh/esp8266_milight_hub/releases/tag/1.10.10-beta2

[VossenTech]

Seems to be working correctly now, thanks.

[sidoh]

Awesome! Thanks for your patience.

I will probably not cut a 1.10.10 release, but this will be included in 1.11.0.