Packets from some remotes (e.g., RGB) are never sniffed

xbmcnut commented 6 years ago

Great work so thank you for this. I intend using it to replace a dead v4 Milight bridge and with Hass.io

I have flashed 1.6.5 onto a NodeMCU V2 and have a NRF24L01 module (that was working with My Sensors previously) and I have managed to pair one MiLight bulb (CCT) and can control on/off, brightness and color temp. Based on this, I'm assuming the NRF module is fully working?

My question is I have two different 2.4Ghz RF remotes; one for a flash pendant wave light and another for some RGB LED strips. Both lights work with their consecutive remotes.

When I activate sniffing though and use either remote, I see nothing. I do see data when I use the UI to control the one paired bulb but nothing when using the remotes. Am I supposed to see something when the remote turns on my lights that are not paired with any bridge?

P.S I have not configured anything in the MQTT section yet other than my broker address.

sidoh commented 6 years ago

Any traffic espMH intercepts while in listening mode should show up. Shouldn't matter if the remotes are paired with anything.

If it's not showing up, it might be a remote model that's not supported. Could you:

Let me know what model the remote is? futlight.com seems to have the best product catalog.
See if there's anything interesting in Serial logs with the DEBUG_PRINTF flag enabled?

xbmcnut commented 6 years ago

Thanks for the prompt reply. The LED RGB strip remote appears to be a FUT098. The other one is not shown at futlight.com but here is a picture.

img_20180428_223906

I'm assuming I'll have to compile to enable the debug flag?

sidoh commented 6 years ago

That's right, build flags are here:

https://github.com/sidoh/esp8266_milight_hub/blob/master/platformio.ini#L28

My guess is that both of those should work. The pictured one looks like a FUT007. Never used one myself, so not positive that it would show up.

xbmcnut commented 6 years ago

I there. Bit rusty on Platformio but tried it with settings below.

Serial monitor inside Platformio have me nothing but 'null null null null' so I connected to COM4 with Putty. Got valid data when I used an actual FUT007 (found the one that came with my bridge), but the remote pictured above and the FUT098 show nothing. Maybe there are just outside the band of the NRF2401 I have?? I have a LT8900 (with NRF2401+ chip) so maybe I'll try that next.

xbmcnut commented 6 years ago

The other module I have is not a LT8900 (as sold to me) but the ultra mini version of the NRF2401+. I hooked it up anyway and get the same results from PuTTy.

*WM: AutoConnect
*WM: Connecting as wifi client...
*WM: Using last saved values, should be faster
*WM: Connection result:
*WM: 3
*WM: IP Address:
*WM: 10.0.1.182
Setup complete
Radio is available

Packet received: E05A58D540B09778EE030303Packet transformed: 075A1AAB020DE91E77C0  C0C0Failed CRC: expected 0, got 49344

Only see these results when pushing buttons on the 'real' FUT007. Nothing from the other remotes. Does that tell us anything?

sidoh commented 6 years ago

Sniffing on the real LT8900 might even be broken. So maybe a good thing it wasn't a real one? :)

Definitely looks like an unsupported protocol. Might even suggest it's knockoff hardware that wasn't implemented to spec.

Could you get a few more packet captures and label them with what button they correspond to?

xbmcnut commented 6 years ago

Thanks but that's the thing. The knock off remotes do not generate any code in PuTTy even though they work on their respective lights.

sidoh commented 6 years ago

Sorry, I should've been more clear. I meant the logs from serial output. Like the stuff you pasted above:

Packet received: E05A58D540B09778EE030303Packet transformed: 075A1AAB020DE91E77C0  C0C0Failed CRC: expected 0, got 49344

This contains the packet data.

xbmcnut commented 6 years ago

Thanks Chris but that is the problem. I have three remotes and only one gives me a data output through PuTTY. Not sure I did the Platformio build right but it did upload and the NoceMCU board works so I'll take that as a win. I'm assuming that if if didn't build with the DEBUG_PRINTF enabled then I'd getting nothing on serial? I can't post anything for the two 'rogue' remotes as nothing is being 'seen' by the NRF2401.

sidoh commented 6 years ago

If you're seeing that "Packet received" line, you successfully compiled with DEBUG_PRINTF.

If you're not seeing serial output for your remotes, then they're probably sending on different channels and with different syncwords. We'd likely need to sniff the SPI bus to get those parameters.

I ordered a FUT007 from ebay, but won't be here for a few weeks.

xbmcnut commented 6 years ago

Thanks for all your help, much appreciated. Is there a way I can set up a sniff from the SPI bus?

sidoh commented 6 years ago

Doing this is quite a bit more heavyweight than most things required for this project. You'd need a logic analyzer or oscilloscope, and you'd probably need to take apart the remote so you could attach probes to the appropriate pins.

xbmcnut commented 6 years ago

I have an oscilloscope and an electronics background and don' like being beaten by a machine! :) If I can help get these remotes working, I'm happy to help if you can point me in the right direction.

sidoh commented 6 years ago

It's not something I actually have any experience with. I think people basically find the microcontroller or the radio, figure out where the SPI pins are, and watch for messages as actions are taken. Fancier scopes can decode SPI messages. Some of the folks who have helped out with this before have digital logic analyzers that do this.

sidoh commented 6 years ago

Got my FUT007. Works with CCT:

cct packet received (7 bytes):
Request type  : 5A
Device ID     : 333A
b1            : 02
b2            : 04
b3            : 2A
Sequence Num. : FE
cct packet received (7 bytes):
Request type  : 5A
Device ID     : 333A
b1            : 02
b2            : 04
b3            : 29
Sequence Num. : FD
cct packet received (7 bytes):
Request type  : 5A
Device ID     : 333A
b1            : 02
b2            : 04
b3            : 28
Sequence Num. : FC

xbmcnut commented 6 years ago

Thanks, I appreciate you going to that effort. One of my three remotes is indeed that one that is branded 'Mi-Light' and I can confirm mine works too. The other two do not however. The colour LED strip one is this one https://www.ebay.com/itm/2-4G-LED-RGB-RGBW-Controller-Touch-Screen-RF-Remote-Control-for-RGBW-LED-Strip-/262983073383?roken=cUgayN&soutkn=huGRwh. Can I send you one of those to try? PM me if that is OK.

sidoh commented 6 years ago

No need, I think. I have the milight wifi bridge which is capable of emulating that remote.

I can confirm I'm not seeing any sniffed traffic on my end.

We'll need to get the radio configs (channels, syncwords) to add support for the remote. This writeup shows a teardown of a milight device. My wifi gateway has both of these chips, but I don't have the equipment to do this myself.

Also possible that someone else has done this already.

sidoh commented 6 years ago

Found a really cheap digital logic analyzer that works great.

Don't have great probes, but managed a hacky solder job of magnet wire to the SPI pins on the radio:

I think there's a bug in the listening logic for the nRF24L01 radio code. The syncwords and channels are already configured, but packets aren't showing up. Seems like some packet types just aren't working.

Thinking it's probably a regression introduced in 1.6 when the nRF code was optimized. Will try to find some time to dig in over the next couple of weeks.

xbmcnut commented 6 years ago

Wow Chris, that is above and beyond! You made my day though :). If I can get my new WiFi dining room light and the pool LED's voice controlled by HA, I can put my feet up with a cold one for at least a week! Cheers for all your hard work, much appreciated. P.S Can you link to the analyzer you got? Might pick one up myself.

sidoh commented 6 years ago

https://www.amazon.com/gp/product/B077LSG5P2

And to think I was halfheartedly considering getting a $400 Saleae.

xbmcnut commented 6 years ago

Wow, that is awesome! If it did the trick, that is all I need too. Thanks.

sidoh commented 6 years ago

Just an update here --

This is definitely an issue with the nRF radio code. I poked around a little bit, but haven't made much progress yet. (I did, however, manage to get the radio configs for all remotes supported by the ibox)

@khmann I recall that some of the radio optimizations you contributed a few months back may have broken sniffing from official devices (this was something we agreed was fine at the time). Does this sound familiar to you?

If so, any pointers on adding this support back without breaking the radio? Definitely happy to investigate myself, but thought I'd ask in case you have any ideas offhand.

xbmcnut commented 6 years ago

Thanks for the update.

xbmcnut commented 6 years ago

Any NRF changes in the esp8266_milight_hub_nodemcuv2-1.8.0-dev1 build?

sidoh commented 6 years ago

@xbmcnut hey, sorry I'm so slow getting back to you. Busy summer!

No, no nRF changes in v1.8. At the rate I'm going it probably won't make it in.

xbmcnut commented 6 years ago

Thanks for the update, appreciated. Let us know once you have something I can test.

khmann commented 6 years ago

Sorry, I dropped off the internet for awhile. The "bug" relates to the not-clearly-defined PL1167 trailer bits; It is possible that these bits are encoded differently by different transmitters and my original patch (RF mod was to replace the rather silly PL1167 software decoding by using nRF hardware filters) had a problem with some of the syncword calculation. I believe this was actually fixed in the second patch, but that there was some other problem that broke everything. I've been meaning to pick this back up (and actually use it this time - previously I was just using the reusing your radio code for my own project)

EDIT: part of the problem is in the below area, I was always confused how to set the "trailer" bits... never found that "defined" to my liking... I haven't looked at it in awhile, maybe I was wrong and it should always be 0x05 or 0x0a . nRF stores syncword backwords and flipped, so that's part of why the whole setup is so goofy. If I get back on this project again, nRF actually supports 2 simultaneous RX syncwords, I'll just add the alternate trailer as well.

[code] if (_syncword0 & 0x01) { _nrf_pipe[ --nrf_address_pos ] = reverse_bits( ( (_syncword0 << 4) & 0xf0 ) + 0x05 ); } else { _nrf_pipe[ --nrf_address_pos ] = reverse_bits( ( (_syncword0 << 4) & 0xf0 ) + 0x0a ); } [/code]

khmann commented 6 years ago

Or I'm confused and that's the preamble (I know I verified the preamble function is correct.) Maybe we need to add the following code (or it's inverse...?) I'm sorry I'm still trying to figure out platformio to play with it here

 -  _nrf_pipe[ --nrf_address_pos ] = reverse_bits( ( (_syncword3 >> 12) & 0x0f ) + 0x50 );  // kh: spi doc says trailer is always "5" ?
 +  if (_syncword3 & 0x8000) {
 +    _nrf_pipe[ --nrf_address_pos ] = reverse_bits( ( (_syncword3 >> 12) & 0x0f ) + 0xa0 ); 
 +  } else {
 +    _nrf_pipe[ --nrf_address_pos ] = reverse_bits( ( (_syncword3 >> 12) & 0x0f ) + 0x50 );
 +  }

I agree the code needs something, but I think it's worth considering the possibility that some things which look milight compatible, quite simply, aren't. I've certainly been bitten by that...

xbmcnut commented 6 years ago

Thanks guys. Let me know if you'd like me to test a couple of different variants with the 'non-working' remotes I have.

xbmcnut commented 6 years ago

Hi there. Any chance this has been improved in 1.8.5?

hhaensel commented 5 years ago

Hi there, I have just tried to sniff commands from my newly acquired FUT098. But no response at all :-( However, I succeeded in pairing my RGBW controller (FUT027) by chosing an arbitrary device ID, selecting the RGB (!) protocol and pressing pair 3s after switching on the controller. I can now control the device via mode/color/brightness/speed. But also the commands sent from the ESP are not sniffed! So I guess the problem is still not solved in "1.8.7"?

xbmcnut commented 5 years ago

Any progress on this issue Chris?

sidoh commented 5 years ago

Hey, sorry for not responding to your earlier comment, @xbmcnut. Slipped through the cracks. No, I've honestly not looked at it. No plans to soon, although if it's important to a number of folks I could prioritize it.

cristianlivella commented 5 years ago

Same problem here with FUT098 remote and FUT025 controller. I can pair and send comands with NRF24L01, but I can't sniff the remote.

sidoh commented 5 years ago

FUT098 is supported.

cristianlivella commented 5 years ago

FUT098 is supported.

It isn't working... during sniffing I see only the packets sent by the NRF24L01.

sidoh commented 5 years ago

Ah, sorry, missed the context in the thread.

Sniffing doesn't work with it. Some weird stuff in radio config I haven't had the interest or time to dig into.

Think it's probably somewhere in this change:

https://github.com/sidoh/esp8266_milight_hub/commit/22f3baf667745f6de8887f0482b10073ef0da7e1#diff-955946430a21b8fcd99b02c8913272db

xbmcnut commented 5 years ago

I'm a patient man. Let us know if you ever get this fixed and I'll give it another try.

xbmcnut commented 5 years ago

Any of the new builds have a fix for this yet Chris?

sidoh commented 5 years ago

No, unfortunately not. If I'm honest, this is not a very fun sort of issue to work on, so sort of hard to drum up the motivation to dig in.

There's probably room in the 1.10 release, but it's possible I won't have the time or motivation to dig through fiddly radio code (apologies for the inconvenience).

xbmcnut commented 5 years ago

Thanks. I'll move on.

sidoh commented 5 years ago

I'd like to keep the issue open. It is definitely an actual bug.

xbmcnut commented 5 years ago

Understood.

sidoh commented 5 years ago

Good news... think I figured this out. :)

Should have a patch up later today.

@khmann called it. I think the PL1167 trailer is different for the RGB remote (the bits are inverted).

xbmcnut commented 5 years ago

Wow, that's great news. Looking forward to testing.

sidoh commented 5 years ago

Notes from poking around with the (A) the radio code, and (B) SPI dumps from the wifi gateway:

The register that defines packet syncword lengths, etc. (0x20) is not set, which means the defaults are used:
- Preamble: 1 byte
- Syncword: 8 bytes (only 0-2 and 4-6 are set. others assume the default of 0)
- Trailer: 4 bits
There does not seem to be a way to set the values for the preamble or trailer. I think this means there's something internal to the PL1167 that generates these values.
Because the values are different based on the syncword, they seem to be a function of the syncword. (why? who knows...)
Seems to depend on the parity of syncword0.
SPI dump when sending an RGB packet: rgb_spi.csv.gz
SPI dump when sending an RGB+CCT packet: rgb_cct_spi.csv.gz
SPI dumps ended up not helping much, but attaching them to show that defaults are not changed.
PL1167 User Manual

What ended up revealing what was going on was reducing the nRF24 address with to 4 (from 5), which showed the packet even if the trailer didn't match. Then you could see what the value of the trailer was for all remote types. Will hardcode these values into the radio configs.

sidoh commented 5 years ago

Fixed in 1.10.0-dev.4.

xbmcnut commented 5 years ago

Hi Chris,

Just tried 1.10.0-dev.4 (nodemcuv2) and sadly it is still not sniffing packets from this remote https://github.com/sidoh/esp8266_milight_hub/issues/266#issuecomment-385165682

sidoh commented 5 years ago

Hm.

I'm testing with the official wifi gateway, and I am able to sniff RGB packets (where I was not able to before).

Is it able to sniff packets from other remotes?

xbmcnut commented 5 years ago

Yes. My original MiLight remote is being sniffed. Just trying to find my 3rd remote to test that.

sidoh / esp8266_milight_hub

Packets from some remotes (e.g., RGB) are never sniffed #266