REQUEST: Hue Entertainment Support

[mog3n]

I think they fixed the delay now, hue effects are nearly instantaneous while connected to my chroma accessories, and while playing overwatch. It's really quite good! Any plans for supporting the new API with screenBloom?

[alexkerin]

This would be really nice.

[yadomi]

This could be great, but not that easy. The Hue Entertainement API use UDP instead of HTTP which is why it's faster. This mean a lot of code rewrite because screenbloom use the http API.

Also, as far as I know, the Entertainement SDK is only available to some members choosen by Philips, it's not entirely public at this time.

[Kakifrucht]

Also, as far as I know, the Entertainement SDK is only available to some members choosen by Philips, it's not entirely public at this time.

Pretty sure that the SDK is already available, however it is in C++. The protocol itself is not really exposed, I haven't done too much research though. Other than that this does not address ZigBee latency between the lights.

[yadomi]

Well, the EDK and SDK links provided here from the Hue documentation are not found on GitHub. Don't know how to access them.

However, I think it shouldn't be that hard to send UDP packet in python anyway

[kershner]

Yo! Hey everyone, somehow I missed this thread. I've been trying to mess with the Entertainment stuff but am currently stuck and haven't really gotten off the ground quite yet. I'll try and write it up here.

Here's a Gist of a little test script I'm using to try and get some Hue Entertainment action going. Quick walkthrough:

1. Hue Bridge needs to be v. 1.22 or greater for Entertainment stuff
2. Entertainment Groups are now a thing. Will make addressing lights easier I think, no more individual bulb shenanigans, just let users decide how they want their lights grouped in the official app.
3. Need to toggle group streaming before any Entertainment business can be sent to the Entertainment Group. Will need to toggle streaming on when ScreenBloom is activated, and toggle it back off again when ScreenBloom is stopped.
4. DTLS Handshake with the Hue Bridge to start sending stream messages.

This is where I'm currently stuck. I haven't been able to successfully DTLS handshake with the bridge. After a small bit of researching I think the problem could be due to a limitation of the Python OpenSSL Windows build, which could prove very problematic for our issue here heh.

In the Entertainment documentation Philips provide a little sample handshake chunk of code written in C++. Problem is we're using Python here for ScreenBloom. Specifically the issue I think comes from the fact that not only is Philips using this new secure DTLS protocol, they're using it with a PSK (pre-shared keys) and a very specific Cipher Suite: TLS_PSK_WITH_AES_128_GCM_SHA256.

Luckily a DTLS library has been developed for Python. And we're also lucky that someone developed PSK support for Python's official SSL module.

The problem I think ultimately comes from the fact that Python's SSL module is built on OpenSSL, and that particular build (possibly ONLY the Windows builds?) do not include the specific cipher suite that Hue Entertainment is using. Here are the list of supported OpenSSL cipher suites, scroll down to CIPHER SUITE NAMES, which would be the strings we would need to pass to sslpsk.wrap_socket().

This is currently my hunch as I haven't had a lot of time to really dive into Windows logs and the like. I wonder if OpenSSL is built differently on Macs vs Windows, and maybe it has additional cipher suites included. That might be where I would look next.

Here's a reddit thread I started on the subject, possibly helpful.

[yadomi]

Hello, I've did some research too and I'm stuck on the same point than you. I've found your reddit post when searching for DTLS stuff.

I can successfully connect using the openssl command line client after setting the entertainement area to active (this can be tricky for testing since it goes back to a disabled state within 10s of no activity as the documentation says)

Here is what am I doing:

HUE_PSK_IDENTITY=x7JgmmATbH4CEkoJtkMHSnuQgHJ1NFFgxArYgz4y
HUE_PSK=ADC6817879F1F796FBF80709B1DFD2C9
./openssl s_client -connect 192.168.0.10:2100 -psk_identity $HUE_PSK_IDENTITY -psk $HUE_PSK -dtls1_2 -cipher PSK-AES128-CBC-SHA
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 278 bytes and written 570 bytes
---
New, TLSv1/SSLv3, Cipher is PSK-AES128-CBC-SHA
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : DTLSv1.2
    Cipher    : PSK-AES128-CBC-SHA
    Session-ID: DC140722B5DAC1DF5BEB810A0311BDA2C3494712A75BF6F3B5BFF7B16DDE0BF5
    Session-ID-ctx:
    Master-Key: C7B86754687A227660A5293D3D8D9B16088E2A3951A59582988598CB10238D4D8752F43661E47402D6B4AC120E50FCA6
    Key-Arg   : None
    PSK identity: x7JgmmATbH4CEkoJtkMHSnuQgHJ1NFFgxArYgz4y
    PSK identity hint: None
    SRP username: None
    Start Time: 1517144147
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

This work on macOS with openssl 1.0.2n and on Windows with 1.0.2l. I think it has to do in how openssl has been installed/compiled on the machine, but I'm not an openssl export anyway.

Does a command like this work on your windows machine ?

[kershner]

Working on it. Looking at yours PSK, I wonder if my PSK credentials are correct. Can you tell me how you got your HUE_PSK from your HUE_PSK_IDENTITY?

[yadomi]

The HUE_PSK is the raw clientkey returned by the JSON on the /api/newdeveloper endpoint with generateclientkey set to true in the payload, I think the s_client do the transform itself

I successfully be able to send UDP packets and changing lights using node with node-dtls-client by setting the clientkey in a hex Buffer.

I will try with your gist to see if I got the same result.

However did you try without toggling the stream right after the dtls_handshake ? (eg here)

[kershner]

Oh, wow. I wonder if that's my issue then. I've been trying to convert the clientkey into the "16 bit binary representation" on this line here. The Entertainment docs explicitly tell you to do that lol, no mention of the client doing the actual work. I will try again with the raw clientkey tonight after work to see if I can get any further.

EDIT - can you post how you structured the UDP message that you're sending to update the lights in an Entertainment group?

[kershner]

Ok, an update.

I was able to successfully DTLS handshake and connect with the OpenSSL Command Line Tool, just like you.

I sent a Hue API command to enable streaming and then quickly fired off this in the cmd prompt: c:\OpenSSL-Win32\bin\openssl s_client -connect 192.168.0.2:2100 -psk_identity 2VmpoTwVvlTN3CwkS0U07jP8Ov5cc9K2Tpjwg5RD -psk 4827F36229025777A194C68FEE87EEA3 -dtls1_2 -cipher PSK-AES128-CBC-SHA

Which spit this out at me, just like yours:

CONNECTED(0000019C)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 282 bytes and written 567 bytes
Verification: OK
---
New, SSLv3, Cipher is PSK-AES128-CBC-SHA
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : DTLSv1.2
    Cipher    : PSK-AES128-CBC-SHA
    Session-ID: 85D1AABA5A71AA2F33F4C1D68B1E2B91376A5C50EF466D0CE4AF28475A30BD2D
    Session-ID-ctx:
    Master-Key: 6199C8186FEE1D7B544DFA1982CC3216851E16D8F753DEB919E28E3ADD045BDAA43B53E69490DEF5926D12EB71311D51
    PSK identity: 2VmpoTwVvlTN3CwkS0U07jP8Ov5cc9K2Tpjwg5RD
    PSK identity hint: None
    SRP username: None
    Start Time: 1517273193
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
closed

So I think this narrows the problem down to either my code or one of the Python libraries, dtls or sslpsk.

I'm leaning toward sslpsk because when I run my entertainment_test.py script I get a big scary official Windows error, 0xC0000005 or Access Violation. Checking the error in the Windows Event Viewer tells me that it was thrown by this file: ..\screenBloom\app\venv\lib\site-packages\sslpsk\SSLEAY32MD.dll, which is a library used by OpenSSL. But this could be because I've written the handshake wrong.

So that's where I currently am. I'm not sure if my sslpsk Python code is bad or if there is a problem with the DLL, or something more general about the OpenSSL that sslpsk was compiled with.

---

Edit - I think I might be leaning more toward pydtls actually. The guy in the reddit thread mentioned this, from the pydtls readme:

Installation of OpenSSL on Microsoft Windows operating systems is inconvenient. For this reason, source distributions of PyDTLS are available that include OpenSSL dll's for 32-bit and 64-bit Windows

So I wonder if I can find a different version of pydtls that will work. I'll update more once I have a chance to dig into that tomorrow/later in the week.

[Drommedhar]

Is there any new news about support for the entertainment API? Getting access to the whole SDK was pretty easy. Just had to fill in some stuff into the form and tell them what I want to develop and after 2-3 days, I got full access.

Have you got around the SSL problem already?

I'm asking as I'm really looking forward to ScreenBloom supporting the "new" API, as all other Applications still are not supporting it either (or at least hueDynamic isn't).

[kershner]

Still stuck on the DTLS Handshake on Windows. Haven't had time to look further into it. 40% sure I'm doing something wrong, 60% sure that a deep change to pydtls or its dependencies will be required.

[alexkerin]

Have you guys seen this? https://www.youtube.com/watch?v=sPH1u7Z7y9E&feature=youtu.be

Successful implementation using the new API - latency seems to be well down.

[Drommedhar]

I'm using it for about a week already. It's still pretty early, so a lot of needed stuff like Autostart or Control over Contrast/Brigthness etc is still to be done. But for what it currently does, it works well.

Have tested it with a stroboscope and there was no delay.

EDIT: Also I can confirm that the developer of hueDynamic will be working on integrating the new API. Asked him about it via email and he said "in some weeks". So this will be coming too.

[kershner]

huestacean and I'm assuming hueDynamic are both written in C++, which makes Entertainment API integration pretty straightforward (Hue's own documentation/examples are in C++). The problem for ScreenBloom is that we're using Python, and I don't think anyone's come up with a nice and tidy way to connect to the Entertainment API with Python tools just yet. All the pieces are there (sslpsk, pydtls) but they don't really work well together, and especially not on Windows which is the main platform for ScreenBloom.

So the options for ScreenBloom at the moment are: port entire program to C++ or dive deep into some obscure Python/OpenSSL/Windows issues and try to figure out what the issue is. Neither are very good options, especially with Hue Sync on the horizon making this whole program obsolete. So since I don't really have enough time to do either, I'm waiting to see if someone in the Python Hue community can make some more progress. If we can get the DTLS thing cracked then adding Entertainment support to ScreenBloom should fall into place pretty easily (with a bunch of work on my end that is hehe).

[ghost]

Has this progressed any further as the diyHue project is also written in python and has got to a similar sticking point to you? Any help would be super appreciated.

[trainman261]

Just wanted to mention that I would still love to see an update to screenbloom to implement the new API. I wouldn't consider it obsolete since Hue Sync doesn't support Windows 7, which many people still use. Essentially, those of us using Windows 7 don't have any options to do this via the newer, more responsive API.

[DankDataViz]

Just wanted to add that I for one would like to NOT see the new API implemented. I am currently using a third party Gledopto controller for controlling my LED strips. These are, however, not compatible with the Hue API. So if Screenbloom would switch over to the Hue API I would no longer be able to use Screenbloom!

[ghost]

Hey guys. I'm not sure how far you got with this but the guys at the diyHue project have made some progress. It looks like they have made a successful connection. Have a look at this comment onwards. I hope it can help you too. One slight caveat, the "breakthrough" is not written in python. It's a separate script, I believe in C, that will handle the entertainment API side of things and the communicate with the existing diyHue code.

[d8ahazard]

Yo! Hey everyone, somehow I missed this thread. I've been trying to mess with the Entertainment stuff but am currently stuck and haven't really gotten off the ground quite yet. I'll try and write it up here.

Here's a Gist of a little test script I'm using to try and get some Hue Entertainment action going. Quick walkthrough:

1. Hue Bridge needs to be v. 1.22 or greater for Entertainment stuff
2. Entertainment Groups are now a thing. Will make addressing lights easier I think, no more individual bulb shenanigans, just let users decide how they want their lights grouped in the official app.
3. Need to toggle group streaming before any Entertainment business can be sent to the Entertainment Group. Will need to toggle streaming on when ScreenBloom is activated, and toggle it back off again when ScreenBloom is stopped.
4. DTLS Handshake with the Hue Bridge to start sending stream messages.

This is where I'm currently stuck. I haven't been able to successfully DTLS handshake with the bridge. After a small bit of researching I think the problem could be due to a limitation of the Python OpenSSL Windows build, which could prove very problematic for our issue here heh.

In the Entertainment documentation Philips provide a little sample handshake chunk of code written in C++. Problem is we're using Python here for ScreenBloom. Specifically the issue I think comes from the fact that not only is Philips using this new secure DTLS protocol, they're using it with a PSK (pre-shared keys) and a very specific Cipher Suite: TLS_PSK_WITH_AES_128_GCM_SHA256.

Luckily a DTLS library has been developed for Python. And we're also lucky that someone developed PSK support for Python's official SSL module.

The problem I think ultimately comes from the fact that Python's SSL module is built on OpenSSL, and that particular build (possibly ONLY the Windows builds?) do not include the specific cipher suite that Hue Entertainment is using. Here are the list of supported OpenSSL cipher suites, scroll down to CIPHER SUITE NAMES, which would be the strings we would need to pass to sslpsk.wrap_socket().

This is currently my hunch as I haven't had a lot of time to really dive into Windows logs and the like. I wonder if OpenSSL is built differently on Macs vs Windows, and maybe it has additional cipher suites included. That might be where I would look next.

Here's a reddit thread I started on the subject, possibly helpful.

Any progress on this? Trying my own implementation...one thing I discovered is that the cipher has to be called as "PSK-AES128-GCM-SHA256", the name listed in the Hue doc is the IANA name.

I'm close, but now sslpsk is complaining that _sslobj can't be found when I try to wrap the socket...

def do_handshake(self):
    print("Handshaking with ", self.bridge_ip)
    sock = False
    psk = binascii.unhexlify(self.bridge_key)
    print("PSK", psk)
    try:
        do_patch()  # dtls library doing its thing
        print("Patched")
        host = self.bridge_ip
        port = 2100
        print("Wrapping socket")
        tcp_socket = socket(AF_INET, SOCK_DGRAM)
        ssl_sock = sslpsk.wrap_socket(tcp_socket,
                                      ssl_version=258,
                                      ciphers='PSK-AES128-GCM-SHA256',
                                      psk=psk, hint=self.user)
        print("Socket wrapped")
        ssl_sock.connect((host, port))
        print("Socket connected")

        msg = "ping"
        ssl_sock.sendall(msg.encode())
        msg = ssl_sock.recv(4).decode()
        print('Client received: %s' % msg)
    except Exception as e:
        print("Socket exception: ", e)
    return sock

[aarongirard]

Hello from 2022. Has anyone made any progress on using the entertainment api (streaming) via python? Running into the same issue with dtls handshaking atm... Thanks!