Interest in bringing iOS remote control capabilities to UxPlay

[thiccaxe]

I was able to successfully pair the remote app on an iPad to a python server. I will continue to develop it so that it is possible to respond to app remote controls server-side (up down, left right, select, etc.)

Just wondering if there is scope to integrate with UxPlay in some way.

[fduncanh]

Quite possibly, but I don't know.

I learned more about AirPlay crypto from the recent work on pair-pin-start, which I had expected to lead to the fully-encrypted communications mentioned by FD- in, but doesn't.

What I implemented (and just added to the master branch) is just the previously-unimplemented parts of "Legacy Pairing" (luckily the pair-verify part that follows was already done in UxPlay, I'm not sure by which previous developer) . I'm not sure who will want to use the pin protocol, but now it's there, anyway.

The "fully-encrypted communication" that FD- talks about is in fact HomeKit pairing, and pair_ap and pyatv seem to have figured out a lot about it and have it working, at least in part.

At some point, if it became necessary, Homekit pairing could probably replace Legacy pairing in UxPlay.

I am still wondering how an Apple client sends the missing SPS and PPS NAL unit that starts high-resolution h265 video data, which it doesn't seem to do in Legacy pairing. Maybe it somehow only does it in Homekit paring? I suppose the way to find out would be by evesdropping on the communications of the high-res client with an AppleTV 4K. "Remote control" mode also seems to be something connected to HomeKit.

[thiccaxe]

Modern "Remote control" (HomeKit/airplay 2 based) and the slightly less modern (MRP based) both use the non legacy encryption.

For now, I used a more "legacy" pairing (which is where the iOS client provides a pin which is entered onto the server) to make the remote. This runs on DAAP. Even this can optionally use FairPlay, but I haven't been able to see any actual encryption happening (nor does the iOS client attempt to use fp with my implementation).

Once I get this remote working, I will upgrade the MRP version, then to the even newer version.

[thiccaxe]

I suppose the way to find out would be by evesdropping on the communications of the high-res client with an AppleTV 4K

Perhaps atvproxy can help out here? Last time I used it, it was a bit unstable.

[thiccaxe]

https://github.com/thiccaxe/DAAPRemoteServer/ has a 95% working server implementation.

To control a client airplaying/mirroring to UxPlay, UxPlay would need to dump the "Active Remote" header and the ip address of the connected client to a file.

The flow would look like:

UxPlay --  (active remote, addr of mirroring client)      ---->  
                                                                                                    tv remote server ----> play/pauses the mirroring client
ios remote in control center --(play/pause commands)  ------>

[fduncanh]

Can you be more specific?

UxPlay would need to (optionally) write (or overwrite) a file just containing the currently-connected client's ip address and "Active Remote" header each time a connection starts?

The first SETUP request seems to have all this plus the client DeviceID

Handling request SETUP with URL rtsp://192.168.1.223/4123947610597705189
DACP-ID: 7305C9B24FD5799E
Active-Remote: 4072375325
SETUP 1

Just this info would allow the remote app to read the files and to control media playing by the client?

[socketa4techx7]

Workaround (bash PoC):

$ cat airtunes-ctrl.sh #!/bin/bash # Airplay specs / commands taken here: https://nto.github.io/AirPlay.html#audio-remotecontrol # Pre-req: uxplay output shout be placed to /home/pi/tmp.txt

ActiveRemoteToken=$(cat /home/pi/tmp.txt | grep Active-Remote | tail -n 1)

# Command below scans mDNS for iOS connected to the uxplay: #avahi-browse --all -p -r -t | grep IPv4 | grep "iTunes Remote Control" | tail -n 1

# avahi-browse --all -p -r -t | grep IPv4 | grep "iTunes Remote Control" | tail -n 1 | cut -d ";" -f 7,8,9 #Example output: SK-iPhone-2.local;192.168.88.253;58050

rawstring=$(avahi-browse --all -p -r -t | grep IPv4 | grep "iTunes Remote Control" | tail -n 1 | cut -d ";" -f 7,8,9)

clientName=$(echo $rawstring | cut -f 1 -d ";") clientIP=$(echo $rawstring | cut -f 2 -d ";") clientPort=$(echo $rawstring | cut -f 3 -d ";")

echo "DEBUGDEBUG: IP PORT NAME = $clientIP $clientPort $clientName"

echo "Executing command: playpause toggle" curl $clientIP:$clientPort/ctrl-int/1/playpause -H "$ActiveRemoteToken" -vv

[socketa4techx7]

Code I've posted above allows to execute command from the list below on the iOS client, connected to uxplay (when music played, or screen shared). This is not ideal, but at least gives an ability to understand what's happening there.

Basically, iOS client connects to uxplay, then registers mDNS record about itself under name "iTunes Remote Control". One of the attributes of this record is management port, which we use. It changes on every connection.

Then we connect to iOS from uxplay host with curl via HTTP to, e.g., 192.168.88.253:58050/ctrl-int/1/playpause and that makes iOS to apply, basically, playpause function.

commands list taken here - https://nto.github.io/AirPlay.html#audio-remotecontrol beginff	begin fast forward
beginrew	begin rewind
mutetoggle	toggle mute status
nextitem	play next item in playlist
previtem	play previous item in playlist
pause	pause playback
playpause	toggle between play and pause
play	start playback
stop	stop playback
playresume	play after fast forward or rewind
shuffle_songs	shuffle playlist
volumedown	turn audio volume down
volumeup	turn audio volume up

[socketa4techx7]

And yes, forgot to mention - Active-Remote token is being changed regularly, but that's not a big deal to get from currently existing uxplay output (with debug options which enriches technical data being posted (-d option)). And we can put this header to our HTTP GET request with curl (curl -H "Active-Remote: 123123123" blablabla)

[fduncanh]

Currently UxPlay with -ca writes (and overwrites) a file with coverart that arrives in audio-only mode. It now also writes a file $HOME/.uxplay.pem with its private key (to maintain a fixed public key for pin-registrants. It would be pretty easy (I guess) to monitor the Active-Remote to and write it and other info like client DeviceID and ip address to a file when uxplay starts or when this info is updated.. (And delete it when UxPlay finishes)

If this is all you need, what data should go in that file? what format?

EDIT I will very soon release UxPlay 1.67.
are you requesting a short delay to add this feature?

[fduncanh]

Ah yes, I see: when connected to UxPlay the client advertises its remote control service

$ avahi-browse -ta
+   eth0 IPv6 iTunes_Ctrl_6***********EDBD                  iTunes Remote Control local
+   eth0 IPv4 iTunes_Ctrl_6***********EDBD                  iTunes Remote Control local

[thiccaxe]

I don't think that we need to necessarily delay 1.67, but maybe (later) some sort of current "statistics" file could give information about the current client? (json?)

• name
• mac
• remote ip address
  • connected ports?
• current encryption keys?
• the Active-Remote header
• anything else?

[fduncanh]

I have already almost finished adding a feature (in branch "remote") that will output to file "$HOME/.uxplay.dacp" (or a file of your choosing) current values of

1. client (remote) ipaddress (ipv4 or ipv6)
2. DACP-ID
3. Active remote header other stuff could be added too.

[thiccaxe]

With https://github.com/thiccaxe/DAAPRemoteServer/commit/a2212aaef8dedc4ebce5d6a35ab422bd17d4a375 if you provide the active remote and DAAP id (ip address is not really needed due to mdns) the playpause functionality works. I'll only implement that for now, since anything further is pretty opionated.

For now the active remote/daap is copy-pasted from uxplay logs.

LMK if you all have any thoughts on how the the controls available could be mapped to the "commands list taken here - https://nto.github.io/AirPlay.html#audio-remotecontrol"

[fduncanh]

@thiccaxe Your feature request is now in uxplay-1.67 (on master branch, not yet a release)

use "-dacp" to output DACP-ID and Active-Remote to $HOME/.uxplay.dacp use "-dacp <path-to-file>" to change the output file

file is transient (created when connection starts, deleted when connection closes)

[fduncanh]

done?

[fduncanh]

@thiccaxe

would you like to test before the 1.67 release (also the pin stuff?)

[thiccaxe]

@fduncanh I added the feature to the latest commit in my repo. Things seems to work as expected, and are mostly stable. @socketa4techx7 / anyone else if you can test on your end that would be nice.

However from the uxplay end of things, I don't think anything blocks a release now. Nice that Uxplay cleans up after itself (deletes the .uxplay.dacp when its done) too!

[thiccaxe]

the pin pairing seems to be working without issues.

[thiccaxe]

Actually, we should probably test what happens in the case that UxPlay doesn't have full read write access to the dacp (or pem) file

[fduncanh]

You are right.

Since UxPlay is run by a user, not as root I assume $HOME is writable, but since the user can change this, a test of the choices is needed. Do this in uxplay.cpp first, if -key fn or -dacp fn are used.

Does FILE *fp f= open(fn, "a") fail if write permission is not granted? would need to test if file "fn" exists first, so it can be deleted if it didn't exist before the test which would then create it.

[thiccaxe]

I will try to test this, likely in a few hours though.

[fduncanh]

https://unix.stackexchange.com/questions/159557/how-to-non-invasively-test-for-write-access-to-a-file might be useful

EDIT maybe not, this is shell script not C

this is C

https://www.delftstack.com/howto/c/c-check-if-file-exists/

access() seems the solution. hope it works on macOS and MINGW64 Windows .

EDIT: here is windows version https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/access-waccess?view=msvc-170

[thiccaxe]

I tested various combinations of the .uxplay.pem file permissions (and permissions of its parent directory) and things seem to work - uxplay gracefully ignores the permission issues and just uses a random key. maybe uxplay should log a warning if it is unable to read/write to the .uxplay.pem.

I'm not on a network where I can test the dacp file but if the code is the same it should work fine.

[fduncanh]

@thiccaxe thanks! uxplay now tests "filename" for write access when the option " -dacp filename" (etc) is given.

EDIT. 1.67 is released