search

sidoh / esp8266_milight_hub

Replacement for a Milight/LimitlessLED hub hosted on an ESP8266
MIT License
934 stars 220 forks source link

B8 wall switch support #95

Closed PetricaM closed 6 years ago

PetricaM commented 7 years ago

Hi,

Could you add support for B4 wall switch (similar to B4)?

https://www.aliexpress.com/item/Mi-Light-B8-8-Zones-Wall-Mount-Smart-RGBW-CCT-Color-temperature-Brightness-adjustment-Panel-Remote/32814525399.html?

sidoh commented 7 years ago

Thanks Petricia!

It sounded like you had some packet captures from the device. Could you include them here?

PetricaM commented 7 years ago

Yes, here it is (only there's no second "i" :) )

Serial monitor reads only gibberish (I've used nodemcu-flasher on Windows). I just realized: I've flashed a v2 image on a v3 Nodemcu so I think this might be the issue for serial monitor not working (changing baud rate has no effect).

Anyway, the physical device works fine with RGB_CCT however not with RGBW so there might be a hardware related issue with the B8.

  1. From associated bulb (RGB_CCT) - group 1 rgb_cct packet received (9 bytes): Raw packet: 19 99 E2 E5 61 40 9C 62 2F

Decoded: Key : 19 b1 : 25 ID : 46D9 Command : 01 Argument : 01 Sequence : 9B Group : 01 Checksum : 43

  1. From Group 6 (not associated):

rgb_cct packet received (9 bytes): Raw packet: 9C 3C BF 78 82 EE 52 83 74

Decoded: Key : 9C b1 : 25 ID : 46D9 Command : 01 Argument : 06 Sequence : 9C Group : 06 Checksum : C1

sidoh commented 7 years ago

Whoops. Thanks Petrica :)

Weird- I've only seen Serial gibberish with mismatched baudrates. It should be 9600. There aren't meaningful differences between v2 and v3, I think. The only thing that matters in the pre-compiled firmware images are flash size configs. I think v2 and v3 both have 4MBs of flash.

This is interesting. b1 is treated as a constant, and normally has the value 0x20, but it's 0x25 from your captures. Guessing this is a protocol identifier.

I think there's another remote that supports 99 groups (link). Wonder if it's compatible with the bulbs as well.

I think the Milight wifi box I have supports both of these remotes. I'll take a closer look when I have more time.

sidoh commented 7 years ago

@PetricaM - could you link what bulbs you're using with the remote? Just double-checked, and my RGB+CCT (FUT015) bulbs don't work with the 8 group remote. Looks like FUT105 are supposed to work with the 8 group remote. Are these the bulbs you have?

PetricaM commented 7 years ago

Hi, I have FUT014 and FUT103. FUT103 can be associated with both B8 and B4 and work fine. FUT014 only with B4. I've set forwarders (B4 to B8 and B8 to B4) in HA to control from one remote the bulbs associated with the other. B4 to B8 bulbs works fine (with all commands), however B8 to B4 bulbs has the colors reversed :D Also when punching white on the B8 it sends other device id than the one that is used for colors. To summarize: B8 cand control FUT014 (with forwarder, not associated) however colors are messed up. I think that setting up virtual devices and associating them with B4 and B8 should work (actually 2 for B8). Any way to fiddle with colors in HA in the forwarder for codes sent by B8? Thanks

PetricaM commented 7 years ago

Hi,

Seller mentioned that B8 is compatible with the following units: FUT103, FUT104, FUTT03, FUTT04, FUTT05, FUT066, FUT015, FUT069.

For FUT015 it might be a typo, though (instead of FUT105).

sidoh commented 7 years ago

Oh interesting.

Is this something you could clarify with them?

I tried unpairing a FUT015 bulb with the remote it's using, and then pairing with a B8 remote (via the Milight wifi box), but it wasn't responsive.

PetricaM commented 7 years ago

Done.

The seller replied with the same list as posted above and the fact that B8 works, in fact, with FUT015...

Actually I couldn't find FUT105 at any Aliexpress shop. I think it hasn't entered mass production yet.

sidoh commented 7 years ago

I tried again to pair a FUT015 bulb with the B8 remote from my Milight app with no luck. The bulb just wasn't responding at all. Also tried pairing a bulb to the same ID (118F) for the usual RGB+CCT remote and controlling with the B8 remote, but no luck.

Here are some packet captures:

rgb_cct packet received (9 bytes):
Raw packet: 46 61 5C 71 86 59 63 46 DC 

Decoded:
Key      : 46
b1       : 25
ID       : 118F
Command  : 01
Argument : 01
Sequence : 9A
Group    : 01
Checksum : CC
rgb_cct packet received (9 bytes):
Raw packet: 64 74 46 F2 BA 25 96 BA C7 

Decoded:
Key      : 64
b1       : 25
ID       : 118F
Command  : 01
Argument : 01
Sequence : 98
Group    : 01
Checksum : 6C
rgb_cct packet received (9 bytes):
Raw packet: FA AD 90 BD DA AD 9C 9A D9 

Decoded:
Key      : FA
b1       : 25
ID       : 118F
Command  : 01
Argument : 01
Sequence : 97
Group    : 01
Checksum : 1D
rgb_cct packet received (9 bytes):
Raw packet: 87 68 49 D1 1B F9 01 2F 04 

Decoded:
Key      : 87
b1       : 25
ID       : 118F
Command  : 01
Argument : 01
Sequence : 96
Group    : 01
Checksum : 89
rgb_cct packet received (9 bytes):
Raw packet: F5 BD 73 93 BD 9C 06 BE B4 

Decoded:
Key      : F5
b1       : 25
ID       : 118F
Command  : 01
Argument : 01
Sequence : 95
Group    : 01
Checksum : 1A
rgb_cct packet received (9 bytes):
Raw packet: 9F 50 D1 89 43 AC 87 D7 CD 

Decoded:
Key      : 9F
b1       : 25
ID       : 118F
Command  : 81
Argument : 0A
Sequence : 94
Group    : 01
Checksum : B8
rgb_cct packet received (9 bytes):
Raw packet: 52 55 58 75 82 5E 68 42 E6 

Decoded:
Key      : 52
b1       : 25
ID       : 118F
Command  : 01
Argument : 0A
Sequence : 93
Group    : 01
Checksum : CA
PetricaM commented 7 years ago

I think that the b1 value of 25 instead of 20 on the B8 affects the way the packets are decoded and, in fact, the device id might be wrong or incomplete.

I can associate the device id sniffed for B8 against both FUT103 and FUT014 however only FUT103 then works from the physical device.

Using a forwarder B4 to B8 (bulb is associated to B8) works fine. When trying B8 to control a bulb associated to B4 the colors are reversed (and the white button turns off the bulb).

I see that the group id is not decoded correctly on B8 (also on White command it sends group id 15 :) ). Except for Groups 1 and 2 OFF command, the pattern is similar however the bias is different for on and off commands.

Group 1 ON -> {"device_id":18137,"group_id":1,"device_type":"rgb_cct","state":"ON"} Group 2 ON -> {"device_id":18137,"group_id":2,"device_type":"rgb_cct","state":"ON"} Group 3 ON -> {"device_id":18137,"group_id":3,"device_type":"rgb_cct","state":"ON"} Group 4 ON -> {"device_id":18137,"group_id":4,"device_type":"rgb_cct","state":"ON"} Group 5 ON -> {"device_id":18137,"group_id":0,"device_type":"rgb_cct","state":"OFF"} Group 6 ON -> {"device_id":18137,"group_id":1,"device_type":"rgb_cct","state":"OFF"} Group 7 ON -> {"device_id":18137,"group_id":2,"device_type":"rgb_cct","state":"OFF"} Group 8 ON -> {"device_id":18137,"group_id":3,"device_type":"rgb_cct","state":"OFF"}

Group 1 OFF -> "device_id":18137,"group_id":1,"device_type":"rgb_cct","command":"mode_speed_up","state":"ON"} Group 2 OFF -> "device_id":18137,"group_id":2,"device_type":"rgb_cct","command":"mode_speed_down","state":"ON"} Group 3 OFF -> "device_id":18137,"group_id":7,"device_type":"rgb_cct","state":"OFF"} Group 4 OFF -> "device_id":18137,"group_id":8,"device_type":"rgb_cct","state":"OFF"} Group 5 OFF -> "device_id":18137,"group_id":9,"device_type":"rgb_cct","state":"OFF"} Group 6 OFF -> "device_id":18137,"group_id":10,"device_type":"rgb_cct","state":"OFF"} Group 7 OFF -> "device_id":18137,"group_id":11,"device_type":"rgb_cct","state":"OFF"} Group 8 OFF -> "device_id":18137,"group_id":12,"device_type":"rgb_cct","state":"OFF"}

Group 0 ON -> "device_id":18137,"group_id":0,"device_type":"rgb_cct","state":"ON"} Group 0 OFF -> "device_id":18137,"group_id":4,"device_type":"rgb_cct","state":"OFF"}

sidoh commented 7 years ago

I wouldn't expect the B8 remote to work as-is, or only partially work if at all.

For the FUT092/T4 (RGB+CCT) remotes, groups are decoded from the command byte. Each group has its own on/off command. On for groups 0-4 have IDs 0-4, off have IDs 5-9 (or something to this effect). From what I can tell, it's 0-7 and 8-15 for B8. So the group IDs are going to be way off.

This is not the way I'd design the protocol, but there are things that break if packets aren't interpreted this way. Namely, the group byte is the group of the last command, which is wrong for on/off commands.

Reversing the decoding required a bit of bruteforce/guesswork (some details on my blog), and I've lost a lot of the context at this point, but from what I remember --

I think some raw packet captures for things that work and things that don't would be helpful. The stuff from MQTT has some information thrown out.

KnappeP commented 7 years ago

Hi, I have the similar Problem as Petrica when using the new LS2 (5 in 1) strip controller. This controller is only compatible with the B8, FUT089 or the app emulating the FUT089.

Sniffing the "on" command via app gives me the following package:

rgb_cct packet received (9 bytes): Raw packet: 7C 5C 5B 32 E2 4A FF DF 73

Decoded: Key : 7C b1 : 25 ID : 02F7 Command : 01 Argument : 02 Sequence : C9 Group : 02 Checksum : 20

Regards Peter

sidoh commented 7 years ago

This is something I'm going to have a hard time resolving myself because I don't have hardware to test with.

I think at the very least, I'll need a bunch of raw packet captures for commands that work, and a bunch (at least 10 each) for commands that don't work.

sidoh commented 7 years ago

Alternatively, I'm happy to point anyone brave enough to add support on their own in the right direction :)

khmann commented 6 years ago

cct7.txt.gz Ah hello sir... I think perhaps I am that person. I have acquired the FUT105 12W RGB-CCT bulb and FUT089 8 Zone remote. I believe this "8 zone" control is a new protocol, shared with LS2 strip controller and the new outdoor lights. I think it's "Protocol 25". I fully understand the old RGBW world, but this is new to me.

I was hoping to decode it with your library, but it seems it's a little more complicated than that. I held the (ALL)ON button until my thumb hurt, it took a really long time until I had all 256 variations (if you exclude bytes 8 and 10... one of those is SEQ I guess).

Perhaps your mad-krypto-skillz can see the pattern : ) The only thing I can see is that fields 3,5,6,7,9,(3+7),(3+9),(5+6) | uniq only use 192 of 256 possible values.

I should add that capture is fully CRC valid and only packets received multiple times in sequence, so the data is good.

EDIT: NVM, git clone sidoh/milight_decoding works fine (thankyou), just need to figure more the protocol.

sidoh commented 6 years ago

Looks like you figured it out, but think this protocol is essentially the same as the RGB+CCT protocol. The closest thing I have to documentation is the packet formatter code here:

https://github.com/sidoh/esp8266_milight_hub/blob/1.5.0/lib/MiLight/RgbCctPacketFormatter.cpp

This blog post also covers the packet structure a little bit in case that's helpful:

http://blog.christophermullins.com/2017/03/18/reverse-engineering-the-new-milightlimitlessled-2-4-ghz-protocol/

The most important difference from the other RF protocols is that commands are encoded with (command, argument) pairs. So to set brightness to a certain value, you send the "brightness" command byte, and set the argument to the desired brightness.

khmann commented 6 years ago

Yeah I was really just confused about how the encryption was applied. Kudos again, btw. Now I have to figure how to make it in python (for now, until I get some other things taken care of). I think (attached) is what you need when you want to start thinking about this protocol. Your decode_packet.rb worked fine, just had to change to command_decoder=...key(command&0x3) to get the fut089's combination cct/saturation slider. Attached is a series of all-off, adjust hue, adjust sat, adjust bright, all-on, adjust, 1 off, adj, 1 on,...

fut089.txt.gz

I'm still waiting for some last-gen RGBCCT stuff, I'm curious to compare the protocol. futlight.com sales literature strongly suggests many bulbs support multiple protocols, but simultaneously? Might get set per address when paired.

My platform is different, so I'm delighted to help reverse but not likely to do much ESP coding on my own.

sidoh commented 6 years ago

I have one of these things, which emulates what I assume is the same RF protocol as the B8. What I don't have is a bulb to test with. Hard to know which things matter and which don't without that.

Looks like Amazon finally has FUT105s. I just ordered one. Should be able to mess around with it.

PetricaM commented 6 years ago

Hi,

I think is possible that FUT089 and B8 don't use, in fact, the same protocol.

RFLink mentions FUT089 support in their latest R47 firmware revision: http://rflink.nl/blog2/

I don't have a FUT089 but I've tried RFLink with B8 and it gets different Device IDs for on and off commands so it's not decoded correctly and I guess it might not be the same protocol after all.

sidoh commented 6 years ago

The packets you supplied back in July look consistent with what I've seen from FUT089 captures. @khmann just provided a bunch. Maybe you could check if there's a difference?

I'd expect there to be some funny business, especially (maybe exclusively) with group on/off commands, but I think the protocol is otherwise the same.

Looks like Amazon is stocking B8s too. Just ordered one as well.

Does RFLink publish their source? Poked around on their website, but all I found is a zip archive that looks out of date. I'm curious if they leveraged my decoding stuff, or if they reversed things on their own.

sidoh commented 6 years ago

Okay, got a FUT105 today (fortunately had one offering free same-day delivery :)

I can confirm that it works with the emulated FUT089 eight-zone remote via my iBox1 WiFi controller. After pairing with the emulated FUT089, I can control the bulb with esp8266_milight_hub as long as the group ID is <= 4. If the group ID is 5-8, I can send commands with group ID = 0.

It's interesting that on/off works reliably with esp8266_milight_hub when the bulb is paired with an 8-zone remote. I think this means it's dynamically using the value of the "protocol byte" to inform how command arguments are interpreted. For the 4-zone remotes, "on" is cmd = 1, args = 0-4, "off" is cmd = 1, args = 5-8. For 8-zone, "on" is cmd = 1, args = 0-8, etc.

As far as I can tell, the packets from FUT089 look identical to the B8 packets that @PetricaM supplied above, but hard to tell without more samples.

Totally possible that other devices are more sensitive to the differences in the 4-zone vs. 8-zone commands, but the FUT105 actually behaves better than I expected it to.

sidoh commented 6 years ago

Got the B8 today. It... makes noise. o_o

The packet structure is identical to FUT089 for every command I've checked: on/off, mode speed, mode, color, white mode, brightness, saturation.

I did, however, notice a couple of additional differences from the FUT092:

Not sure why I didn't catch these differences before. Just easier having physical devices to test with, I guess. :)

I had hoped the differences would be minor enough that it wouldn't warrant a different protocol, but they are definitely fundamentally incompatible protocols.

I'll target support for this in the next release.

sidoh commented 6 years ago

First pass at FUT089 support in a branch:

https://github.com/sidoh/esp8266_milight_hub/tree/FUT089

It was quite a bit more difficult to add support than I anticipated. FUT089 shares the same radio settings as FUT092, but has a different packet structure. The previous code grouped radio config and packet structure together, and those needed to be separated for both sends and receives to behave as expected.

I tested this out a bit myself, and everything except for modes seems to work presently. Will merge the feature branch into v1.6.0 branch when these tasks are completed.

@PetricaM @khmann @KnappeP would any of you guys be able to test with the FUT089 branch and let me know how it works?

khmann commented 6 years ago

Maybe someday... I haven't a NodeMCU, maybe you've seen a Raspberry Pi fork out there? Im hoping soon to release my own platform with some additional documentation, but there are multiple good reasons to separate RF/syncword from protocol. I've been waiting for some 1st generation RGB bulbs and old remotes from limitlessLED so I could finish testing. I realize a "scene macro" performance advantage because I can start sending commands on an alternate frequency before the trigger command even stops... avoid collisions.

An example, (v4) RGBW light all accept RGBW (Protocol Bx) commands when sent with RGB freq/sync. (They even kind-of support RX on RGBCCT setup, but I can't get many packets through like that, only 1-2/sec, probably my bug, I didn't try much)

We also know we need protocol "21" for the new CCTv2 controllers B2 and FUT091. I'm probably going to buy some LS2 in a couple months...

Surely we are OCD enough to consider support for all the things? Consider the milight PIR (happyttt.com) and clone units (on 2.4G). I wonder what would happen if we looked at the 433MHz strips? Bet it's a same old protocol...

PetricaM commented 6 years ago

Hi Chris,

I would be glad to help testing with B8. However I screwed up the entire PlatformIO config and I don't seem to find time to reinstall the full machine. Can you post compiled bin files of this beta version as the NodeMCU flasher still works?

Thanks

sidoh commented 6 years ago

Sure thing, pre-compiled release here:

https://github.com/sidoh/esp8266_milight_hub/releases/tag/1.6.0-dev1

PetricaM commented 6 years ago

As I don't have FUT105 (B4 and B8 remotes, FU103 and FUT014 bulbs) I've set the following automation to forward packets from B8 to B4.

Actually it worked also pre v.1.6 however device ids and colors were not recognized correctly.

While the color with the forwarder works as expected, there are two issues I've encountered:

I've added MQTT messages (in includes both 1.6 and 1.5).

Thanks

- alias: MiLightForwarder1
  initial_state: True
  trigger:
    platform: mqtt
    topic: 'milight_gw/update/0x46D9/fut089/1'
  action:
    - service: mqtt.publish
      data_template:
        topic: "milight_gw/0x1111/rgb_cct/1"
        payload_template: >
          {{ trigger.payload }}

Colors: Client mosqsub|7658-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/fut089/1', ... (78 bytes)) <= correct topic (from v1.6) {"device_id":18137,"group_id":1,"device_type":"fut089","hue":302,"state":"ON"} Client mosqsub|7658-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/rgb_cct/1', ... (79 bytes)) <= incorrect topic (from v1.5) {"device_id":18137,"group_id":1,"device_type":"rgb_cct","hue":168,"state":"ON"} Client mosqsub|7658-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/0x1111/rgb_cct/1', ... (78 bytes))<= forwarded to correct device {"device_id":18137,"group_id":1,"device_type":"fut089","hue":302,"state":"ON"} <=correct color Client mosqsub|7658-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x1111/rgb_cct/1', ... (68 bytes)) {"device_id":4369,"group_id":1,"device_type":"rgb_cct","state":"ON"} Client mosqsub|7658-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x1111/rgb_cct/1', ... (68 bytes)) {"device_id":4369,"group_id":1,"device_type":"rgb_cct","state":"ON"} Client mosqsub|7658-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x1111/rgb_cct/1', ... (78 bytes)) {"device_id":4369,"group_id":1,"device_type":"rgb_cct","hue":302,"state":"ON"}

White mode: Client mosqsub|8222-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/rgb_cct/15', ... (71 bytes)) <= from 1.5 {"device_id":18137,"group_id":15,"device_type":"rgb_cct","state":"OFF"} Client mosqsub|8222-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/rgb_cct/15', ... (71 bytes)) {"device_id":18137,"group_id":15,"device_type":"rgb_cct","state":"OFF"} Client mosqsub|8222-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/fut089/1', ... (91 bytes)) <= from 1.6 {"device_id":18137,"group_id":1,"device_type":"fut089","command":"white_mode","state":"ON"} Client mosqsub|8222-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/rgb_cct/15', ... (71 bytes)) {"device_id":18137,"group_id":15,"device_type":"rgb_cct","state":"OFF"} Client mosqsub|8222-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/0x1111/rgb_cct/1', ... (91 bytes)) <=correct topic {"device_id":18137,"group_id":1,"device_type":"fut089","command":"white_mode","state":"ON"} Client mosqsub|8222-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x1111/rgb_cct/1', ... (68 bytes)) {"device_id":4369,"group_id":1,"device_type":"rgb_cct","state":"ON"}

Saturation Client mosqsub|7983-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/fut089/1', ... (85 bytes)) {"device_id":18137,"group_id":1,"device_type":"fut089","color_temp":370,"state":"ON"} Client mosqsub|7983-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/rgb_cct/1', ... (96 bytes)) {"device_id":18137,"group_id":1,"device_type":"rgb_cct","button_id":7,"argument":0,"state":"ON"} Client mosqsub|7983-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/rgb_cct/1', ... (96 bytes)) {"device_id":18137,"group_id":1,"device_type":"rgb_cct","button_id":7,"argument":0,"state":"ON"} Client mosqsub|7983-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/rgb_cct/1', ... (96 bytes)) {"device_id":18137,"group_id":1,"device_type":"rgb_cct","button_id":7,"argument":0,"state":"ON"} Client mosqsub|7983-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x46D9/fut089/1', ... (85 bytes)) {"device_id":18137,"group_id":1,"device_type":"fut089","color_temp":370,"state":"ON"} Client mosqsub|7983-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/0x1111/rgb_cct/1', ... (85 bytes)) {"device_id":18137,"group_id":1,"device_type":"fut089","color_temp":370,"state":"ON"} Client mosqsub|7983-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x1111/rgb_cct/1', ... (68 bytes)) {"device_id":4369,"group_id":1,"device_type":"rgb_cct","state":"ON"} Client mosqsub|7983-raspberryp received PUBLISH (d0, q0, r0, m0, 'milight_gw/update/0x1111/rgb_cct/1', ... (68 bytes)) {"device_id":4369,"group_id":1,"device_type":"rgb_cct","state":"ON"}

sidoh commented 6 years ago

saturation on the B8/1.6 triggers color temp; with v1.5 and/or B4 when setting a color and using saturation it works fine, however on 1.6 it sends color temp.

Ah, I should've mentioned this. For intercepting RF packets, only one of saturation or color temp will work right now. It should be available when the big feature I have planned for 1.6 works (state tracking).

It probably worked in 1.5 because it was just forwarding the raw command/arg bytes. That's probably what I should do until stack tracking works.

white mode doesn't reach bulb;

Just to make sure I'm following what's happening here:

  1 'milight_gw/update/0x46D9/rgb_cct/15' {"device_id":18137,"group_id":15,"device_type":"rgb_cct","state":"OFF"}
  2 'milight_gw/update/0x46D9/rgb_cct/15' {"device_id":18137,"group_id":15,"device_type":"rgb_cct","state":"OFF"}
  3 'milight_gw/update/0x46D9/fut089/1'   {"device_id":18137,"group_id":1,"device_type":"fut089","command":"white_mode","state":"ON"}
  4 'milight_gw/update/0x46D9/rgb_cct/15' {"device_id":18137,"group_id":15,"device_type":"rgb_cct","state":"OFF"}
  5 'milight_gw/0x1111/rgb_cct/1'         {"device_id":18137,"group_id":1,"device_type":"fut089","command":"white_mode","state":"ON"}
  6 'milight_gw/update/0x1111/rgb_cct/1'  {"device_id":4369,"group_id":1,"device_type":"rgb_cct","state":"ON"}

It seems like maybe the messages for 1.5 and 1.6 overlap. Is that expected? Messages (1), (2), and (4) should be from 1.5. 1.6 should not interpret the packet resulting in (4) as rgb_cct, but 1.5 would.

Messages (3), (5), and (6) appear to be the correct sequence:

  1. intercept "white_mode" command from B8
  2. HASS forwards message (3) to topic milight_gw/0x1111/rgb_cct/1
  3. send command corresponding to message (5), and send update.

Is the problem you're noticing that (6) doesn't contain {"command":"white_mode"}? I would at least expect that it'd contain a {"color_temp":153} or something to that effect. Odd that it's not showing up. I'll see if I can reproduce.

In the meantime, could you flash with this firmware? Same code, but recompiled with MQTT_DEBUG and DEBUG_PRINT flags. Would be helpful to get serial logs when you press white mode on B8.

firmware-1.6.0-dev1-debug.bin.zip

sidoh commented 6 years ago

Maybe someday... I haven't a NodeMCU, maybe you've seen a Raspberry Pi fork out there? Im hoping soon to release my own platform with some additional documentation, but there are multiple good reasons to separate RF/syncword from protocol. I've been waiting for some 1st generation RGB bulbs and old remotes from limitlessLED so I could finish testing. I realize a "scene macro" performance advantage because I can start sending commands on an alternate frequency before the trigger command even stops... avoid collisions.

I've seen / heard of a few that run on a Pi, but not used them myself. Would rather run this on a dedicated device.

An example, (v4) RGBW light all accept RGBW (Protocol Bx) commands when sent with RGB freq/sync. (They even kind-of support RX on RGBCCT setup, but I can't get many packets through like that, only 1-2/sec, probably my bug, I didn't try much)

We also know we need protocol "21" for the new CCTv2 controllers B2 and FUT091. I'm probably going to buy some LS2 in a couple months...

Surely we are OCD enough to consider support for all the things? Consider the milight PIR (happyttt.com) and clone units (on 2.4G). I wonder what would happen if we looked at the 433MHz strips? Bet it's a same old protocol...

Yeah, definitely happy to support more devices. Really just need people to want support :)

The iBox2 supports every protocol I've come across, including that of the B8/FUT089. Would imagine the new CCT protocol is on there too.

khmann commented 6 years ago

I look forward to fuzzing one. I agree that a "thin" controller is important; I desperately want macros but I don't want to use an automation controller for that and I still can't decide how much state I can get away with keeping in an MCU. I've put up a project page, love to find a way to collaborate.

https://github.com/khmann/milightx/

PetricaM commented 6 years ago

Hi Chris,

I'll flash the new version on weekend.

Is the problem you're noticing that (6) doesn't contain {"command":"white_mode"}? I would at least expect that it'd contain a {"color_temp":153} or something to that effect. Odd that it's not showing up. I'll see if I can reproduce.

That's right, I was expecting the white mode command.

I don't know if it was documented (or if I didn't made a confusion between my NodeMCUs boards): I flashed 1.60 on an older NodeMCU that had been previously connected to my wifi (it was flashed with another sketch, for some analogue sensors). To my surprise it had connected directly to wifi (also webpage was already loaded).

sidoh commented 6 years ago

Great, sounds good.

I don't know if it was documented (or if I didn't made a confusion between my NodeMCUs boards): I flashed 1.60 on an older NodeMCU that had been previously connected to my wifi (it was flashed with another sketch, for some analogue sensors). To my surprise it had connected directly to wifi (also webpage was already loaded).

This is a feature of the ESP8266. It stores wifi credentials in a reserved portion of flash that's not overwritten when you flash new firmware. If you want to clear it, you can use esptool.py's erase_flash command.

PetricaM commented 6 years ago

Hi,

I've flashed the 1.6.0 dev.5. Selecting FUT089 packets option now works with the FUT103 light bulbs using the original associated RGB+CT (there's no need for a new association).

There was no radical difference in behavior compared to the other gateway versions. Usually (95% of the times) the on command works fine, however for off command there's a lower percentage of success (about 75%).

However, after playing around for 5 minutes with the B8 remote forwarding commands to FUT014, the result was that all my MiLight gateways (3) were knocked out. There was no ping response and neither when connected to serial there was no message when I replicated this. I think it is due to the effect of repeating each command too many times, I'm using 3 different MiLight gateways as having only one it cannot reach all light bulbs. There's reinforced iron in the walls and it affects the wireless coverage. For 2.4Ghz wifi there's no effect however for 5Ghz (either N or AC) the signal drops significantly. The upside is, however, that I can hang pictures on the walls without drilling, just by using small magnets :)

I've tried to set packet repeats at multiple values (1, 5, 10 and 50) however, there's a rebel scum :) every now and then that doesn't turn off or on alongside with the other bulbs in the group. This happens for FUT103 groups with two or three bulbs in the group (FUT014 bulb groups work fine). I've solved this by the followings:

With this setup a bulb might miss one or two off commands and stay up for about 15 or 30 minutes longer but it will eventually turn off.

Other stuff that I've found since starting using your gateway 5 months ago:

khmann commented 6 years ago

I've tried to set packet repeats at multiple values (1, 5, 10 and 50) however, there's a rebel scum :) every now and then that doesn't turn off or on alongside with the other bulbs in the group. This happens for FUT103 groups with two or three bulbs in the group (FUT014 bulb groups work fine).

There's a new radio library in the works which might help with those rogue devices (more better retries), but a possible problem in your environment could be collisions if multiple gateways TX at once. As there are so many warnings other places, make sure you have proper (capacitor) decoupling on the nRF power if you have cheap modules. Bad power really can = nRF slow PLL lock time = therefore trouble keeping up with and responding to an MCU.

but where is the bug- radio related (switching syncwords, the bulbs and controller environment) or the controller packet handling...?

maybe try a workaround: since old bulbs work fine... do the FUT103 support the old v4 RGBW commands as FUT014? I was pleasantly surprised that FUT105 works this way, unlike FUT015, I can use same ID with encrypted or unencrypted commands. They are equivalent for on/hue. Brightness is a reduced "resolution" 26 steps instead of 100(?), color temp and saturation are "relative" 11 steps instead of 100(?), off only exists for zones 1-4 and all.

I know is not a "solution", the new remotes are so nice ;)

sidoh commented 6 years ago

Thanks for testing out, @PetricaM!

I've flashed the 1.6.0 dev.5. Selecting FUT089 packets option now works with the FUT103 light bulbs using the original associated RGB+CT (there's no need for a new association).

There was no radical difference in behavior compared to the other gateway versions. Usually (95% of the times) the on command works fine, however for off command there's a lower percentage of success (about 75%).

Yes, I noticed that my FUT105s respond to the FUT092 and FUT089 protocols for the same device ID. Pretty interesting.

Right. As long as your devices worked with the FUT092 protocol, the only real difference would be that you can use all 8 groups, and that B8/FUT089 packets will be properly decoded.

However, after playing around for 5 minutes with the B8 remote forwarding commands to FUT014, the result was that all my MiLight gateways (3) were knocked out. There was no ping response and neither when connected to serial there was no message when I replicated this. I think it is due to the effect of repeating each command too many times, I'm using 3 different MiLight gateways as having only one it cannot reach all light bulbs. There's reinforced iron in the walls and it affects the wireless coverage. For 2.4Ghz wifi there's no effect however for 5Ghz (either N or AC) the signal drops significantly. The upside is, however, that I can hang pictures on the walls without drilling, just by using small magnets :)

It'd be helpful to get a Serial trace, or at least some way to reproduce the problem. The 1.6.0-dev5 branch includes state tracking, which is likely to have a bug or two. :)

I've tried to set packet repeats at multiple values (1, 5, 10 and 50) however, there's a rebel scum :) every now and then that doesn't turn off or on alongside with the other bulbs in the group. This happens for FUT103 groups with two or three bulbs in the group (FUT014 bulb groups work fine). I've solved this by the followings:

I use 50 for my setup. Sending a packet takes 150-200ms with this value, but for me it strikes the right balance.

for NodeMCU (I'm using v3 only) the procedure for restarting the gateway involves removing the 3V pin prior to restarting (either by presing Reset or disconnecting power) and then connecting it again once fully restarted. Case of a power failure, the NodeMCU gateway remains offline. Though it might be an issue with the batch that I have (I've tested with three different units in the same batch).

Interesting. It might be that there's not a proper pull-up/pull-down resistor on the GPIOs the nRF is using. You could try using a different pin.

I think there's an issue with the voltage regulator as removing the gateway (WeMos mini) from 5V microusb and connecting it to serial has resulted in three NRF24 burned (with two different laptops); there was no issue with the gateway itself. Again, there were no issues changing the 5V microusb to another source (non-pc). Good thing NRF24 are cheap and come in large batches :)

Huh. Not noticed this. I use a D1 Mini. Think this is the datasheet for the regulator:

http://www.ysksemi.com/Upload/file/ME6211Series_E4_0.pdf

using either webpage, HA (MQTT JSON or Limitless) or mobile app (tested with the iOS MiLight v3) for changing colors, the first 3-4 fast commands work fine, however after that there is no reply from bulbs; for the B4 there are only occasional misses and I haven't seen missing two commands in a row.

Would be helpful to get some Serial logs here. Do the ESPs lock up / go unreachable?

PetricaM commented 6 years ago

Would be helpful to get some Serial logs here. Do the ESPs lock up / go unreachable?

There is no reply from the gateway for several seconds (webpage is still active however lights do not react). If, eventually, it starts working again several of the last commands are run at once.

This is the only message that appeared when connection the WeMos gateway over serial and using the gateway's webpage (about 7 or 8 times and I couldn't pinpoint it to a specific command). Received request with untracked session ID: 0

I haven't setup a mqtt_state_topic_pattern.

On a second note: is it possible to start light bulbs with 100% saturation (like blue light only, without having also the white light leds)?

I've setup six different scripts (for the (i) daytime, (ii) sunset, (iii) after 00:00 to sunrise and two modes - for (a) CT and (b) RGB) with input sliders for R, G, B and also Color Temp, Dimmer and Saturation. Of these, only saturation doesn't work with either RGB or CT.

I use scripts instead of scenes as in scenes HA doesn't allow for input sliders. Now I'm able to move the sliders in HA frontend or the FloorPlan and it immediately affects the color and intensity of the light.

Thanks

PetricaM commented 6 years ago

@khmann

but where is the bug- radio related (switching syncwords, the bulbs and controller environment) or the controller packet handling...?

I don't speak Korean or Mandarin and I only understand a few Russian words :)) I now have three gateways forming a mesh network which I found to be almost ideal if not for the occasional mishaps (radio packets pass through two walls however through the third one will only pass seldom). The groups with FUT103 are of either 2 or 3 bulbs at distances of 30 cm to 1 m between them so it is not a distance issue (also having the gateway in the same room). I thought about buying a ibox2 to second the gateway for FUT103 light bulbs using Limitless component in HA however I've read that they are quite unreliable.

maybe try a workaround: since old bulbs work fine... do the FUT103 support the old v4 RGBW commands as FUT014? I was pleasantly surprised that FUT105 works this way, unlike FUT015, I can use same ID with encrypted or unencrypted commands. They are equivalent for on/hue. Brightness is a reduced "resolution" 26 steps instead of 100(?), color temp and saturation are "relative" 11 steps instead of 100(?), off only exists for zones 1-4 and all.

Unfortunately FUT103 didn't react to RGBW (they only work with RGB+CT and FUT089).

sidoh commented 6 years ago

There is no reply from the gateway for several seconds (webpage is still active however lights do not react). If, eventually, it starts working again several of the last commands are run at once.

This is the only message that appeared when connection the WeMos gateway over serial and using the gateway's webpage (about 7 or 8 times and I couldn't pinpoint it to a specific command).

Received request with untracked session ID: 0

Think that error message is from the UDP servers. Probably shouldn't be relevant here. Although if you're not using the UDP protocol, probably means your ESP is wasting cycles maintaining those sockets.

I was able to reproduce this by flooding the MQTT topic with ~10 messages in quick succession.

$ for i in $(seq 0 100); do mosquitto_pub -h mymqtt -t 'milight2/0x118d/rgb_cct/1' -m "{\"hue\":$i}";  done

This is pretty far outside my area of expertise, but I suspect what's happening is the MQTT broker (mosquitto in my case) is overwhelming the ESP with TCP retransmits. I suppose the best way to confirm this would be a tcpdump on the server.

Unfortunately there's not an amazing solution to this problem. Probably the best we can do is queue MQTT messages to be processed rather than processing them as they're received.

A crappy but (hopefully) cheap way to prevent this client-side is to rate limit updates. I seemed to run into problems starting around 30 packets all at once.

On a second note: is it possible to start light bulbs with 100% saturation (like blue light only, without having also the white light leds)?

I've setup six different scripts (for the (i) daytime, (ii) sunset, (iii) after 00:00 to sunrise and two modes - for (a) CT and (b) RGB) with input sliders for R, G, B and also Color Temp, Dimmer and Saturation. Of these, only saturation doesn't work with either RGB or CT.

I use scripts instead of scenes as in scenes HA doesn't allow for input sliders. Now I'm able to move the sliders in HA frontend or the FloorPlan and it immediately affects the color and intensity of the light.

I think the only way to enter color mode on the bulbs I use (FUT105, FUT015) is to send a "hue" command. The bulb will have whatever saturation value it last had. I don't think there's a way to change that without being in color mode. You can send, for example, {"hue":0,"saturation":100}, but that'll send a hue packet followed by a saturation packet. So the bulb will briefly have whatever saturation value it had before.

PetricaM commented 6 years ago

I've tried to set packet repeats at multiple values (1, 5, 10 and 50) however, there's a rebel scum :) every now and then that doesn't turn off or on alongside with the other bulbs in the group. This happens for FUT103 groups with two or three bulbs in the group (FUT014 bulb groups work fine).

I think I found the culprit. I swapped the light bulbs for each group and found that the ones that are causing problems are not influenced by their relative position so I guess it is a hardware issue with the bulbs.

I think the only way to enter color mode on the bulbs I use (FUT105, FUT015) is to send a "hue" command. The bulb will have whatever saturation value it last had. I don't think there's a way to change that without being in color mode. You can send, for example, {"hue":0,"saturation":100}, but that'll send a hue packet followed by a saturation packet. So the bulb will briefly have whatever saturation value it had before.

My bad: MQTT JSON light doesn't have saturation. Sending hue (not supported by MQTT JSON light component) does the trick, however rgb always places saturation below 100. '{"state":"on","color":{"b":51,"g":24,"r":89},"saturation":100}' would actually get saturation to 73. As there is white_value which is useless for MiLight, I made an automation that forwards white_value to saturation. However, with rgb, it won't get saturation to 100. And there's also the flickering effect with this :) as there are two messages sent to the bulb.

If useful to anyone, the action part of the two different automations (for RGB/CT) is below - "d" suffix is for day (I've set also for sunset and night :) ).

- alias: White2Saturation
  initial_state: True
  trigger:
    - platform: mqtt
      topic: 'milight_gw/+/rgb_cct/+'
  condition: 
    condition: template
    value_template: '{{ "white_value" in trigger.payload }}'  
  action:
    - service: mqtt.publish
      data_template:
        topic: "milight_gw/{{ trigger.topic.split('/')[-3] }}/rgb_cct/{{ trigger.topic.split('/')[3] }}"
        payload_template: '{ "saturation": {{ trigger.payload_json.white_value }} }'
- service: light.turn_on
  data_template:
    entity_id: light.living
    brightness: '{{ states.input_slider.dimmerd.state | int }}'
    white_value: '{{ states.input_slider.saturationd.state | int }}'
    rgb_color: ['{{states.input_slider.rd.state | int}}','{{states.input_slider.gd.state | int}}','{{states.input_slider.bd.state | int}}']

- service: light.turn_on
  data_template:
    entity_id: light.living
    brightness: '{{ states.input_slider.dimmerd.state | int }}'
    color_temp: '{{ states.input_slider.colord.state | int }}'
    white_value: '{{ states.input_slider.saturationd.state | int }}'
sidoh commented 6 years ago

Glad you found the rebel bulb :)

Ah, I see what you're saying. Cool automation! I can't think of a better way offhand.

Btw - created #124 for the issue you mentioned about lockups. Unfortunately probably not totally solvable, but can probably improve things quite a bit.

fab33 commented 6 years ago

I just buy some FUT103 GU10 bulbs and I have exctly same problem. Sometimes some bulbs inside the group doesn't respond to the command. All my other milight bulbs (E27) are working fine. It's quite strange, becasue the problem occurs frequently with 2 bulbs and rarely with others (3 at this time)

PetricaM commented 6 years ago

Hi,

I (partly) solved the problem by resending the same command twice in HA with 100 ms delays between commands. Using a second MiLight hub also clears some of the problems but the issue with the hubs is that they tend to loose wifi connection and the ESP8266 needs to be manually reset (I use a RF plug to automatically shutdown power and then restart after a few moments) thus I think that a wired connection (such as Wemos D1 R2 and a W5100 ethernet plug) would be the ultimate gateway for MiLight :)

However, sometimes the bulbs just don't respond. For some of these that are not responsive I have done a reset and for a while they work fine but afterwards the issues comes back.

In some cases the bulbs respond to an original MiLight remote but some bulbs do not respond even to this, so I think it is a hardware problem.

Among the other best practices mentioned by Chris that I've tried I found adding a capacitor between vin and ground helped a little but the issues are not completely gone.

Anyway, I think that (in addition to the different compatible protocols compared to FUT014), the design of the GU10 is faulty as the heat accumulates inside the bulb as the E27 variant works fine.

I'm curious about FUT105 :)

fab33 commented 6 years ago

I have 2 FUT105. Works very well. I never have failure like FUT103. All FUT103 have some problem for some bulbs it's only 5% and for other it's 75% of failure. I have order an ibox to see if i have more reliable results.