[Companion] Documentation Dumps

thiccaxe commented 5 months ago

What to investigate?

With my goal being to setup a companion server so that one can use built-in remote app to control any arbitrary interface (for example, use it as a mouse with the touch pad, or just control media), I thought it prudent to essentially dump any logs and progress I might encounter here, notably anything that is missing in pyatv (even though it may not be useful to add such a thing...)

note, on latest 17.3 beta ios

Captions

related to #720

Enable

Set Captions Settings (13)

{'_i': '_mcc', '_x': 0, '_btHP': False, '_c': {'_mcs': 4, '_mcc': 13}, '_t': 2}

Get Captions Settings (12)

{'_i': '_mcc', '_x': 0, '_btHP': False, '_c': {'_mcc': 12}, '_t': 2} -> {'_c': {'_mcs': 2}, '_t': 3, '_x': 0}

Disable

Set Captions Settings (13)

{'_i': '_mcc', '_x': 0, '_btHP': False, '_c': {'_mcs': 4, '_mcc': 13}, '_t': 2}

Get Captions Settings (12)

'_i': '_mcc', '_x': 0, '_btHP': False, '_c': {'_mcc': 12}, '_t': 2} -> {'_c': {'_mcs': 3}, '_t': 3, '_x': 0}

NO CLUE how this works, seems to be unlike what is described in 720. Sending the set caption settings (same message each time) seems to toggle _mcs from 2 and 3. Has 0 effect in Netflix, youtube, zee

Note that the _mcf in a video is 1806. In this case bits 0x0040 and 0x0080 are set to 0, whereas in music for example mcf is 511, in which both bits are set to 1. Related?

mcf while watching a youtube ad is 270, 256 while switching from one ad to the next ad. reverts to 1806 in regular operation. 1805 when you go to home screen, then 256 a second later.

Siri Remote bluetooth

Related to new feature where you can find Siri/Apple TV remote from ios

Get keys/info about siri remote.

Not sure about behavior when there 2+ remotes connected

{'_i': 'FetchSiriRemoteInfo', '_x': 0, '_btHP': False, '_c': {}, '_t': 2}

{'_c': {'SiriRemoteInfoKey': BPLIST}, '_t': 3, '_x': 0}

BPLIST:

{
    "$version": 100000,
    "$archiver": "NSKeyedArchiver",
    "$top": {
        "root": UID(1)
    },
    "$objects": [
        "$null",
        {
            "serialNumber": UID(0),
            "tvModel": UID(4),
            "tvName": UID(3),
            "supportsFindMy": True,
            "btAddressData": UID(6),
            "irkData": UID(7),
            "firmwareVersion": UID(8),
            "identifier": UID(5),
            "connected": True,
            "$class": UID(9),
            "name": UID(2)
        },
        'Serial?', 'Apple TV name', 'Apple tv model', 'uuid', '6 bytes', '16 bytes', '0x83',
        {
            "$classname": "TVRCSiriRemoteInfo",
            "$classes": [
                "TVRCSiriRemoteInfo",
                "NSObject"
            ]
        }
    ]
}

'6 bytes' is mac address.

tangentially related to discussion in #720 ? It seems that a NSObject is wrapped in a Bplist is wrapped in a OPACK frame...

Events

{'_i': '_interest', '_x': 0, '_c': {'_regEvents': ['PushSiriRemoteInfo']}, '_t': 1}

same as fetch. sent on airpods connect for some reason.

Enable searching:

{'_i': 'ToggleFindingMode', '_x': 0, '_btHP': False, '_c': {'FindingModeEnabledKey': 'YES'}, '_t': 2}

Disable searching:

{'_i': 'ToggleFindingMode', '_x': 0, '_btHP': False, '_c': {'FindingModeEnabledKey': 'NO'}, '_t': 2}

Supported Actions events

Register:

{'_i': '_interest', '_x': 0, '_c': {'_regEvents': ['SupportedActions']}, '_t':

GuideSupportedKey

seems to be updated on any major window change (state changes (power on, screensaver off), entering the quick menu (press and hold home button) and entering/exiting apps). related to accesibility?

{'_i': 'SupportedActions', '_x': 0, '_c': {'GuideSupportedKey': False}, '_t': 1}

NowPlayingInfo

seems to be somewhat handled in MRP protocol.

Register:

{'_i': '_interest', '_x': 0, '_c': {'_regEvents': ['NowPlayingInfo']}, '_t': 1}

Play

Note that is NOT A GUARANTEE that video/audio is being played or not. for example, netflix preview triggers this. zee preview does not trigger this.

{'_i': 'NowPlayingInfo', '_x': 4159442948, '_c': {'NowPlayingInfoKey': BPLIST}, '_t': 1}

BPLIST: netflix tv show, and

{
    '$version': 100000,
    '$archiver': 'NSKeyedArchiver',
    '$top': {'root': UID(1)},
    '$objects': ['$null', {'$class': UID(3), 'playbackRate': UID(2)}, 1.0, {'$classname': 'TVRCNowPlayingInfo', '$classes': ['TVRCNowPlayingInfo', 'NSObject']}]
}

apple music song

{
    '$version': 100000,
    '$archiver': 'NSKeyedArchiver',
    '$top': {'root': UID(1)},
    '$objects': ['$null', {'$class': UID(3), 'playbackRate': UID(2)}, 1.0, {'$classname': 'TVRCNowPlayingInfo', '$classes': ['TVRCNowPlayingInfo', 'NSObject']}]
}

have same. There is no difference between audio and video, seemingly, to tv remote app. note _mcf bit field

Pause

BPLIST

{
    '$version': 100000,
    '$archiver': 'NSKeyedArchiver',
    '$top': {'root': UID(1)},
    '$objects': ['$null', {'$class': UID(3), 'playbackRate': UID(2)}, 0.0, {'$classname': 'TVRCNowPlayingInfo', '$classes': ['TVRCNowPlayingInfo', 'NSObject']}]
}

Touch interface

it is likely critical to figure out how NS is calculated before this could be used.

_tPh values: 5: "click" action; tap, press and hold, has (ended?) 1: swipe started 3: swipe progressing 4: swipe ended

x, y is cx, cy. x ranges left to right, 0-1000 y ranges top to bottom, 0-1000.

Swipe situation

simply send the hidT (timestamped?) with _ns and the appropriate x and y coordinates as the swipe happens. send the appropriate _tPh values as well. In case that there is more complex situations you need to deal with, see below.

hidT will not recieve OPACK ack from server. hidC functions as you would expect.

Tap

quickly aftger each other {'_i': '_hidC', '_x': 0, '_btHP': False, '_c': {'_hBtS': 1, '_hidC': 6}, '_t': 2} {'_i': '_hidC', '_x': 0, '_btHP': False, '_c': {'_hBtS': 2, '_hidC': 6}, '_t': 2} {'_i': '_hidT', '_x': 0, '_c': {'_ns': 41004766480958, '_tFg': 1, '_cx': 1000, '_tPh': 5, '_cy': 1000}, '_t': 1} x and y as _cx _cy, will always bee 1000,1000 no matter where it is actually pressed

Press and hold

quickly after each other 1) {'_i': '_hidC', '_x': 0, '_btHP': False, '_c': {'_hBtS': 1, '_hidC': 6}, '_t': 2} 2) {'_i': '_hidT', '_x': 0, '_c': {'_ns': timestamp? big number at least 46 bits, maybe 64?, '_tFg': 1, '_cx': 1000, '_tPh': 5, '_cy': 1000}, '_t': 1} x and y as _cx _cy, will always bee 1000,1000 no matter where it is actually pressed

release

{'_i': '_hidC', '_x': 0, '_btHP': False, '_c': {'_hBtS': 2, '_hidC': 6}, '_t': 2

Press and hold then swipe to right

1) {'_i': '_hidT', '_x': 0, '_c': {'_ns': 40673733253583, '_tFg': 1, '_cx': 335, '_tPh': 1, '_cy': 611}, '_t': 1} 2) {'_i': '_hidT', '_x': 0, '_c': {'_ns': 40674033085625, '_tFg': 1, '_cx': 340, '_tPh': 3, '_cy': 598}, '_t': 1} 3) {'_i': '_hidC', '_x': 0, '_btHP': False, '_c': {'_hBtS': 1, '_hidC': 6}, '_t': 2} -> recieved ack for 3, not sure if it is needed to wait for this? 4) {'_i': '_hidT', '_x': 0, '_c': {'_ns': 40674033085625, '_tFg': 1, '_cx': 340, '_tPh': 5, '_cy': 598}, '_t': 1} 5) {'_i': '_hidC', '_x': 0, '_btHP': False, '_c': {'_hBtS': 2, '_hidC': 6}, '_t': 2}

as you can see, x changed from 335 to 340

dial action

the physical remote supports circular motion at the border of its touch surface to wind forward/backwards. This is not supported in ios apple tv remote app in general.

tap with two+ fingers.

regular tap but tph is set to 4. Does no action in tvos, but you could still detect it I guess.

Pressing the power button, mute button

regular hidC button call {'_i': '_hidC', '_x': 0, '_btHP': False, '_c': {'_hBtS': 1, '_hidC': NUMBER}, '_t': 2} {'_i': '_hidC', '_x': 0, '_btHP': False, '_c': {'_hBtS': 2, '_hidC': NUMBER}, '_t': 2}

Power button: NUMBER = 19 Mute button: NUMBER = 18

Skip forward and backward

{'_i': '_mcc', '_x': 1725146213, '_btHP': False, '_c': {'_skpS': SKIP AMOUNT IN SECONDS AS A FLOAT, '_mcc': 7}, '_t': 2}

examples: 10.0, -10.0

press and hold left arrow button

returns to home screen.

Expected outcome

written above is pretty much everything that I saw missing in pyatv.

Based on this I will begin work on my server. I hope to make updates to pyatv as time allows (but then I have to do testing and make sure my code is good quality and other silly things like that /s)

thiccaxe commented 3 months ago

Apparently there is no need to do pair-verify if pair-setup has just been successfully completed. https://github.com/thiccaxe/CompanionGames/issues/2

I doubt this will be viable in pyatv due to how the "pairing" session is treated seperately from the "connect" session. However, I guess this repo has a bit more visibility in case anyone runs across this!

thiccaxe commented 3 months ago

TVSystemStatus event is used instead of SystemStatus.

thiccaxe commented 3 months ago

Old ios (12) seems to allow siri of all things through the companion remote.

{'_i': '_siriStart', '_x': 4007573237, '_c': {}, '_t': 2}

Record the time this was sent.

then the device sends

{'_i': '_siA', '_x': 1680017961, '_c': {3: bytes, 4: timestamp, relateive to siristart, 5: [{1: 41, 2: 0}, {1: 40, 2: 41}, {1: 42, 2: 81}, {1: 40, 2: 123}, {1: 38, 2: 163}]}, '_t': 1}

then stop the voice input {'_i': '_siriStop', '_x': 4007573235, '_c': {}, '_t': 2}

Likely a solution for #170 , that issue links to protobufs that document what goes inside the 3: bytes field. Not sure what 5 is. Might have something to do with the format? Are not the same on each _siA packet, so they could be a checksum? but they are similar? Could be markers for where the data should be, do not know.

cherpake commented 2 months ago

I have been investigating touchpad too. Having some trouble to fully mimic iOS remote - especially swipe up to close apps. Other than that seems like I managed to make it work. Although I'm seeing another issue - Apple TV closes the socket from time to time, for no apparent reason.

thiccaxe commented 2 months ago

Having some trouble to fully mimic iOS remote - especially swipe up to close apps.

I can take a look at it from my end. Is your remote public?

cherpake commented 2 months ago

No, not public - it's iOS app to control Mac/PC and Apple TV, and I'm just adding touchpad for Apple TV to it - so not even released. I checked your game and now see that you running a sever and controlling the game using iOS Apple TV remote. I'm working from another end :) trying to be a client that control it.

thiccaxe commented 2 months ago

Someone found that double clicking up will close apps when in the app switcher.

cherpake commented 2 months ago

Using _tPh value 5?

thiccaxe commented 2 months ago

No, I believe it's just two sets of press-unpress

thiccaxe commented 2 months ago

Although I'm seeing another issue - Apple TV closes the socket from time to time, for no apparent reason.

Yes. I see this as well, in atvproxy. I am investigating if anything unusual happens when closing apps now.

Having some issues with atvproxy.

cherpake commented 2 months ago

Was able to fix it, Apple TV closing the socket, by sending _systemInfo frame.

thiccaxe commented 1 month ago

@cherpake how are you calculating the _ns value?

thiccaxe commented 1 month ago

It seems that siri has worked its way into the latest update? not sure exactly. You can trigger siri using homebutton press and hold on phones with home buttons and the power button on phones without homebutton. I couldn't run the avtproxy just yet, but idevicelogs tells me this:

homebutton presses are now sent as an hidC_ packet (I assume power button presses are sent as well?)
events for start, stop, hold are sent from what I can tell
_siriStart sent after the hold hidC (is a session, like touchStart)
rapportd requests wifid to switch to a wifi network? possibly the same one appletv is connected to? not really sure. Either that, or if for some reason the phone is connected to the appleTV through bluetooth it wants to switch to wifi? idrk
I assume it sends the same siri data as previously mentioned
_siriEndpoint request — sent from appletv. Must indicate ack of the end of a siri request. (ios responds with {'_x': ---, '_em': 'No request handler', '_ec': 58822, '_ed': 'RPErrorDomain', '_c': {}, '_t': 3})
_siriStop sent at some point

unfortunately the mangling that

postlund / pyatv