Support/Implement WebSocket events for State Changes

keatontaylor commented 5 years ago

Currently amazon supports WebSockets for sending events on certain state changes. This includes media player play/pause/volume/etc events along with last-triggered echo events when a request is made.

This would be extremely valuable for getting real-time dynamic updates based on state changes within the Alexa echo-system and should be reasonably straight forward to implement now that we have the Auth token from a successful login.

I'm going to play around with this and report back. If successful even if we cannot decode the inner WebSocket payload, an event over WebSockets should be sufficient for us to trigger an update.

alandtse commented 5 years ago

I'm not too familiar with WebSockets. Will this give us push functionality so we can avoid polling?

keatontaylor commented 5 years ago

That's the goal, looks like amazon is pushing events over web sockets for:

Player State Change (playing, paused, content change, etc)
Wake word events (a push notification over web sockets whenever any echo device hears the wake word, would be an excellent way for us to trigger an update for last called.)

So in theory this could eliminate all polling.

keatontaylor commented 5 years ago

Looks like this is absolutely possible and the messages can be properly decoded with little effort. Here is the output from a sample program I wrote.

Screen Shot 2019-03-13 at 6 56 44 PM

Looks like each time that someone talks to an echo device a message is pushed to connected WebSocket clients. This is the "PUSH_ACTIVITY" you see in the screenshot. You can also see other push messages for different messages.

Each message does contain the deviceID that is being affected by the player state change or the device that initiated the activity. Getting PUSH_ACTIVITY can be used to gather information from the last_called alexa device and other messages can directly update media players or give us a hint on when to fetch/pull information like track, playlist, etc when things change.

@alandtse what do you think?

alandtse commented 5 years ago

This looks good. Can you implement websockets in alexapi in a parallel set of functions so we can fallback to the http? Or does this require switching to a purely async model? I'm not planning to do any major work in the API for a little bit if you need it stable.

keatontaylor commented 5 years ago

Currently I want to do as minimal changes as possible. Specifically, we can instantiate a WebSocket class within the device alexa_setup() method and simply get a callback for messages from the WebSocket code. This will be entirely one way, only digesting events over the WebSocket and not sending any commands to amazon, those will still be handled with the http RESTful calls.

Basically, once we have a callback with a particular message, when can use the same mechanism that is used to update the devices with the last called to fire an HA event that can be consumed by the individual media players to update their state.

alandtse commented 5 years ago

I agree we can use WS for listen only as that's the biggest issue to solve.

So we're on the same page, the current architecture has HA __init__.py performing spawning an AlexaLogin class to perform the login. Then on successful login setup_alexa() will use AlexaAPI to populate data into the hass.data structure, including the successful AlexaLogin object and pass control to media_player.py to spawn AlexaClient devices based on hass.data. Those spawned AlexaClients then internally use the AlexaLogin object to access AlexaApi objects to interact with Amazon.

Is your thought that during setup_alexa() we'll also spawn the websocket object for all the AlexaClient objects to subscribe to?

I think that should work, but wonder if architecturally it makes sense to define a websocket object outside alexaapi. Or perhaps I'm misunderstanding you on that point.

alandtse commented 5 years ago

@keatontaylor alternatively, if you share the sample program code, I may have some time to integrate it in this weekend.

keatontaylor commented 5 years ago

@alandtse uploading it as a new class in a branch on GitLab once I get the code cleaned up a bit. Look for it in an hour or so.

alandtse commented 5 years ago

Oh awesome! Looking forward to it.

keatontaylor commented 5 years ago

Not perfect, but should work: https://gitlab.com/keatontaylor/alexapy/tree/websocket-notify

Create an instance of WebSocket_EchoClient with the login session, and a python function you'd like to be called when a message is received.

The Message and Content class describe the contents of each message.

alandtse commented 5 years ago

@keatontaylor I think there's something wrong with your cookie building routine. You're not actually using cookie in your for loop. https://gitlab.com/keatontaylor/alexapy/blob/websocket-notify/alexapy/alexawebsocket.py#L32

        for cookie in self._cookies:
            cookies += cookies + "; "
        cookies = "Cookie: " + cookie
        url += url + str(cookies['ubid-main'])
        url += "-" + str(int(time.time())) + "000"

The initial error is that ubid-main is not a valid key for a string. I assume you wanted self._cookies['ubid-main'] since that's a dict. I've converted that section, but still get a SSL error cause I'm not sure of the cookie structure.

EDIT: Looks like urls should be fixed too.

        cookies = "dp-gw-na-js "
        for cookie in self._cookies:
            cookies += cookie + "; "
        cookies = "Cookie: " + cookies
        url +=  str(self._cookies['ubid-main'])

Note, I blanked out my ubid-main value below, but otherwise untouched.

--- request header ---
GET /?x-amz-device-type=ALEGCNGL9K0HM&x-amz-device-serial=wss://dp-gw-na-js.amazon.com/?x-amz-device-type=ALEGCNGL9K0HM&x-amz-device-serial=134-0082565-00*****-1552776909000 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: dp-gw-na-js.amazon.com
Origin: http://dp-gw-na-js.amazon.com
Sec-WebSocket-Key: Yi898f+BTJ5SEBqhfLEaBQ==
Sec-WebSocket-Version: 13
Cookie: dp-gw-na-js x-main; session-id; session-id-time; sid; ubid-main; sess-at-main; session-token; at-main; csrf;

-----------------------
--- response header ---
HTTP/1.1 101 Switching Protocols
Upgrade: WebSocket
Connection: Upgrade
Sec-WebSocket-Accept: FDkVGAgzs30Y2qvYhfJRu0YPYDU=
-----------------------
send: b'\x82\x9de9+\xd0UA\x12\xe9\x01\rM\xe7TX\x0b\xe0\x1d\t\x1b\xe0U\t\x1b\xe1\x01\x19j\xea-m~\x9e '
send: b"\x82\xfe\x00\x9cg\xa8H\xdaW\xd0)\xec\x01\x9e)\xe3R\x99h\xea\x1f\x98x\xeaW\x98x\xe3\x04\x883\xf8\x17\xda'\xae\x08\xcb'\xb6)\xc9%\xbfE\x92j\x9b]\xe0j\xf6E\xd8)\xa8\x06\xc5-\xae\x02\xda;\xf8]\xd3j\x9b\x0b\xd8 \xbb7\xda'\xae\x08\xcb'\xb6/\xc9&\xbe\x0b\xcd:\xf4\x15\xcd+\xbf\x0e\xde-\x8d\x0e\xc6,\xb5\x10\xfb!\xa0\x02\x8ar\xf8V\x9ej\xf6E\xe9$\xaa\x0f\xc9\x18\xa8\x08\xdc'\xb9\x08\xc4\x00\xbb\t\xcc$\xbf\x15\x86%\xbb\x1f\xee:\xbb\x00\xc5-\xb4\x13\xfb!\xa0\x02\x8ar\xf8V\x9ex\xeaW\x8a5\xa73\xfd\x06\x9f"
error from callback <function WebSocket_EchoClient.__init__.<locals>.<lambda> at 0x7f934e42d598>: [Errno 32] Broken pipe
2019-03-16 22:55:10 ERROR (Thread-3) [websocket] error from callback <function WebSocket_EchoClient.__init__.<locals>.<lambda> at 0x7f934e42d598>: [Errno 32] Broken pipe
  File "/root/venv/lib/python3.5/site-packages/websocket/_app.py", line 345, in _callback
    callback(self, *args)
  File "/root/venv/lib/python3.5/site-packages/alexapy/alexawebsocket.py", line 47, in <lambda>
    ws),
  File "/root/venv/lib/python3.5/site-packages/alexapy/alexawebsocket.py", line 143, in on_open
    ws.send(self._encodeWSHandshake(), OPCODE_BINARY)
  File "/root/venv/lib/python3.5/site-packages/websocket/_app.py", line 153, in send
    if not self.sock or self.sock.send(data, opcode) == 0:
  File "/root/venv/lib/python3.5/site-packages/websocket/_core.py", line 253, in send
    return self.send_frame(frame)
  File "/root/venv/lib/python3.5/site-packages/websocket/_core.py", line 278, in send_frame
    l = self._send(data)
  File "/root/venv/lib/python3.5/site-packages/websocket/_core.py", line 448, in _send
    return send(self.sock, data)
  File "/root/venv/lib/python3.5/site-packages/websocket/_socket.py", line 151, in send
    return _send()
  File "/root/venv/lib/python3.5/site-packages/websocket/_socket.py", line 136, in _send
    return sock.send(data)
  File "/usr/lib/python3.5/ssl.py", line 869, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 594, in write
    return self._sslobj.write(data)
send: b"\x88\x82-'\x1a\x1d.\xcf"
2019-03-16 22:55:10 ERROR (Thread-3) [alexapy.alexawebsocket] WebSocket Error [SSL: BAD_LENGTH] bad length (_ssl.c:1949)

alandtse commented 5 years ago

The error is caused by the _encodeWSHandshake() call I believe.

keatontaylor commented 5 years ago

I'll have to take a closer look at this, I didn't immediately test with a valid cookie once in the class, but will.

alandtse commented 5 years ago

Do you have a spec or example you're using for how to build the handshake?

keatontaylor commented 5 years ago

Branch on gitlab updated. handshake and message parsing is pulled from the OpenHAB Alexa component: https://github.com/openhab/openhab2-addons/blob/master/addons/binding/org.openhab.binding.amazonechocontrol/src/main/java/org/openhab/binding/amazonechocontrol/internal/WebSocketConnection.java

alandtse commented 5 years ago

Ok, I think I have a working version. Will probably push it with some other updates after some more testing.

alandtse / alexa_media_player

Support/Implement WebSocket events for State Changes #119