search

FDH2 / UxPlay

AirPlay Unix mirroring server
GNU General Public License v3.0
1.53k stars 78 forks source link

Work in Progress for airplay video streaming (was: Alternative Implementation. #134

Closed thiccaxe closed 3 months ago

thiccaxe commented 1 year ago

It seems that this implementation of airplay protocol (https://github.com/alexfansz/AirplayServer-1) has all of the airplay protocol implemented (and is compatible with windows). It just needs a front end.

EDIT (2024/06/24) : this code is gone now, but can be recognized as an early version of code now released https://github.com/air-display/apsdk-public

Just posting this here, if anyone from the community wants to add such a front end to this. It also has cleaned up code and what not. I am currently testing this software and may ultimately create a frontend implementation (as it is only a library)

thiccaxe commented 8 months ago

:facepalm: forgot to submit the pr, just had the draft

fduncanh commented 8 months ago

The demo used to call renderers which were removed 3 years ago.

https://github.com/FDH2/apsdk-public/commit/26bd8e22fc122eca075ac376336a087ff010e5e6

This could show how to add some back, using gstreamer, for testing.

fduncanh commented 8 months ago

@thiccaxe This is the source just before the renderer (es-player) was removed 3 years ago. The commit is called "fixed all bugs". Work after that is Android and removing working parts of demo.

https://github.com/FDH2/apsdk-public/tree/c516d1d4c5d275aecbd1a63a985efee97ba6697a

I almost got it working, maybe you can

It needs SDL2 https://wiki.libsdl.org/SDL2/Introduction and the ffmpeg libraries libavcodec, libavutil, libavformat and libswscale devel files On my system these had include files in /usr/include/ffmpeg, not in /usr/include/ (which the code required) so I had to do

cd /usr/include
sudo ln -s ffmpeg/libavcodec libavcodec
...

third-party/CMakeLists.txt is already setup to use fairplay.

Now in the root directory, ./generate_linux_proj.sh and then cmake . -DBUILD_APS_DEMO and `make

For me this builds everything, including third-party/libplayfair.a

But at the very end it fails, with some playfair linking issue I haven't been able to figure out. Maybe some cmake issue. I checked that libplayfair.a contains playfair.c.o

[  1%] Building C object third-party/CMakeFiles/curve25519.dir/curve25519/curve25519-donna.c.o
[  3%] Linking C static library libcurve25519.a
[  3%] Built target curve25519
[  5%] Building C object third-party/CMakeFiles/opensslaes.dir/opensslaes/aes_cbc.c.o
[  6%] Building C object third-party/CMakeFiles/opensslaes.dir/opensslaes/aes_cfb.c.o
[  8%] Building C object third-party/CMakeFiles/opensslaes.dir/opensslaes/aes_core.c.o
[ 10%] Building C object third-party/CMakeFiles/opensslaes.dir/opensslaes/aes_ctr.c.o
[ 11%] Building C object third-party/CMakeFiles/opensslaes.dir/opensslaes/aes_ecb.c.o
[ 13%] Building C object third-party/CMakeFiles/opensslaes.dir/opensslaes/aes_misc.c.o
[ 15%] Building C object third-party/CMakeFiles/opensslaes.dir/opensslaes/aes_ofb.c.o
[ 16%] Linking C static library libopensslaes.a
[ 16%] Built target opensslaes
[ 18%] Building C object third-party/CMakeFiles/hlsparser.dir/hlsparser/hlsparse.c.o
[ 20%] Building C object third-party/CMakeFiles/hlsparser.dir/hlsparser/parse_playlist.c.o
[ 21%] Building C object third-party/CMakeFiles/hlsparser.dir/hlsparser/parse_tags.c.o
[ 23%] Building C object third-party/CMakeFiles/hlsparser.dir/hlsparser/parse_values.c.o
[ 25%] Building C object third-party/CMakeFiles/hlsparser.dir/hlsparser/path.c.o
[ 26%] Building C object third-party/CMakeFiles/hlsparser.dir/hlsparser/utils.c.o
[ 28%] Building C object third-party/CMakeFiles/hlsparser.dir/hlsparser/write.c.o
[ 30%] Linking C static library libhlsparser.a
[ 30%] Built target hlsparser
[ 31%] Building C object third-party/CMakeFiles/playfair.dir/playfair/hand_garble.c.o
[ 33%] Building C object third-party/CMakeFiles/playfair.dir/playfair/modified_md5.c.o
[ 35%] Building C object third-party/CMakeFiles/playfair.dir/playfair/omg_hax.c.o
[ 36%] Building C object third-party/CMakeFiles/playfair.dir/playfair/playfair.c.o
[ 38%] Building C object third-party/CMakeFiles/playfair.dir/playfair/sap_hash.c.o
[ 40%] Linking C static library libplayfair.a
[ 40%] Built target playfair
[ 41%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/add_scalar.c.o
[ 43%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/fe.c.o
[ 45%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/ge.c.o
[ 46%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/key_exchange.c.o
[ 48%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/keypair.c.o
[ 50%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/sc.c.o
[ 51%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/seed.c.o
[ 53%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/sha512.c.o
[ 55%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/sign.c.o
[ 56%] Building C object third-party/CMakeFiles/ed25519.dir/ed25519/verify.c.o
[ 58%] Linking C static library libed25519.a
[ 58%] Built target ed25519
[ 60%] Building CXX object src/CMakeFiles/aps.dir/ap_config.cpp.o
[ 61%] Building CXX object src/CMakeFiles/aps.dir/ap_server.cpp.o
[ 63%] Building CXX object src/CMakeFiles/aps.dir/aps.cpp.o
[ 65%] Building CXX object src/CMakeFiles/aps.dir/crypto/ap_aes.cpp.o
[ 66%] Building CXX object src/CMakeFiles/aps.dir/crypto/ap_crypto.cpp.o
[ 68%] Building CXX object src/CMakeFiles/aps.dir/network/xtxp_connection_base.cpp.o
[ 70%] Building CXX object src/CMakeFiles/aps.dir/network/xtxp_message.cpp.o
[ 71%] Building CXX object src/CMakeFiles/aps.dir/mdns/net_service.cpp.o
[ 73%] Building CXX object src/CMakeFiles/aps.dir/mdns/Linux/net_service_impl.cpp.o
[ 75%] Building CXX object src/CMakeFiles/aps.dir/service/ap_airplay_service.cpp.o
[ 76%] Building CXX object src/CMakeFiles/aps.dir/service/ap_casting_content_parser.cpp.o
[ 78%] Building CXX object src/CMakeFiles/aps.dir/service/ap_casting_event_connection_manager.cpp.o
[ 80%] Building CXX object src/CMakeFiles/aps.dir/service/ap_casting_media_data_store.cpp.o
[ 81%] Building CXX object src/CMakeFiles/aps.dir/service/ap_casting_media_http_service.cpp.o
[ 83%] Building CXX object src/CMakeFiles/aps.dir/service/ap_mirroring_audio_stream_service.cpp.o
[ 85%] Building CXX object src/CMakeFiles/aps.dir/service/ap_mirroring_timing_sync_service.cpp.o
[ 86%] Building CXX object src/CMakeFiles/aps.dir/service/ap_mirroring_video_stream_service.cpp.o
[ 88%] Building CXX object src/CMakeFiles/aps.dir/utils/logger.cpp.o
[ 90%] Building CXX object src/CMakeFiles/aps.dir/utils/plist.cpp.o
[ 91%] Building CXX object src/CMakeFiles/aps.dir/utils/utils.cpp.o
[ 93%] Linking CXX shared library ../output/bin/libaps.so
[ 93%] Built target aps
[ 95%] Building CXX object demo/esplayer/CMakeFiles/esplayer.dir/es_player.cpp.o
[ 96%] Linking CXX static library libesplayer.a
[ 96%] Built target esplayer
[ 98%] Building CXX object demo/CMakeFiles/aps-demo.dir/aps-demo.cpp.o
[100%] Linking CXX executable aps-demo
ld: ../output/bin/libaps.so: undefined reference to `playfair_decrypt(unsigned char*, unsigned char*, unsigned char*)'
collect2: error: ld returned 1 exit status
make[2]: *** [demo/CMakeFiles/aps-demo.dir/build.make:104: demo/aps-demo] Error 1
make[1]: *** [CMakeFiles/Makefile2:325: demo/CMakeFiles/aps-demo.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
thiccaxe commented 8 months ago

Probably can solve it by just including FairPlay files instead of the FairPlay project as whole

thiccaxe commented 8 months ago

Why is the git history almost 150 mb? did you see anything weird?

fduncanh commented 8 months ago

where do you see git history size?

I am working on a branch "older" which got rid of everything after the truncation of the demo

thiccaxe commented 8 months ago

du -h root/.git

fduncanh commented 8 months ago

this is from the original air-display github, not my fork

du -h .git
64K     .git/hooks
0       .git/branches
4.0K    .git/info
4.0K    .git/refs/heads
0       .git/refs/tags
4.0K    .git/refs/remotes/origin
4.0K    .git/refs/remotes
8.0K    .git/refs
153M    .git/objects/pack
0       .git/objects/info
153M    .git/objects
4.0K    .git/logs/refs/remotes/origin
4.0K    .git/logs/refs/remotes
4.0K    .git/logs/refs/heads
8.0K    .git/logs/refs
12K     .git/logs
153M    .git
thiccaxe commented 8 months ago

Got to the same playfair error, will try to work on it.

fduncanh commented 8 months ago

still not fixed.

To test:

git clone http://github.com/FDH2/apsdk-public
cd apsdk-public
git checkout older
./generate_linux_proj.sh
cd .build.linux
make

somehow, the demo is not getting linked to playfair.

thiccaxe commented 8 months ago

nm -D --defined-only output/bin/libaps.so

it looks like only the asio library is getting included. Only thing that seperates asio from the rest is the seperate make file?

thiccaxe commented 8 months ago

maybe it's this in root cmakelists? I think it might just be.

# Include paths
include_directories(
    src
    third-party
    third-party/asio/include
)
fduncanh commented 8 months ago

the libs from third-party are all static lib*.a

grep -R playfair_decrypt
Binary file src/CMakeFiles/aps.dir/crypto/ap_crypto.cpp.o matches
Binary file third-party/CMakeFiles/playfair.dir/playfair/playfair.c.o matches
Binary file third-party/libplayfair.a matches
Binary file output/bin/libaps.so matches
fduncanh commented 8 months ago

It wont be the include dirs. problems with those would show up as compiler errors, not linker errors

fduncanh commented 8 months ago

nm -D --defined-only libaps.so wont see the symbols of third-party libs that libaps.so accesses externally. 

nm -D libaps.so | grep playfair
                 U _Z16playfair_decryptPhS_S_
thiccaxe commented 8 months ago

ah yeah. I see the same ---More confused!

fduncanh commented 8 months ago

UxPlay also produces a static libairplay.a, libllhttp.a and libplayfair.a, that get put into the uxplay executable. maybe the CMakeLists.txt there will show what is needed.

look in UxPlay/build/lib

thiccaxe commented 8 months ago

nm -gDC output/bin/libaps.so | grep fair

my thought is that the fairplay functions ends up in the object file, but the linker doesn't know where the binary code to include is. Maybe?

fduncanh commented 8 months ago

In UxPlay, lib , lib/playfair, lib/llhttp are all below the root when uxplay.cpp is built.

In apsdk, src, demo and third-party are parallel to each other.

fduncanh commented 8 months ago 
add_executable( uxplay uxplay.cpp )
target_link_libraries( uxplay
                   renderers
                   airplay
                   )
thiccaxe commented 8 months ago

https://code.whatever.social/questions/39866262/ld-undefined-reference-despite-library-found-and-symbols-exported#39868170

fduncanh commented 8 months ago

It would be easier if libaps were static.

thiccaxe commented 8 months ago

changed it to static. Same error though.

thiccaxe commented 8 months ago 
/usr/bin/ld: ../output/lib/libaps.a(ap_crypto.cpp.o): in function `aps::ap_crypto::fp_decrypt(unsigned char const*, unsigned char*)':
ap_crypto.cpp:(.text+0xcb0): undefined reference to `playfair_decrypt(unsigned char*, unsigned char*, unsigned char*)'
collect2: error: ld returned 1 exit status
make[2]: *** [demo/CMakeFiles/aps-demo.dir/build.make:104: demo/aps-demo] Error 1
make[1]: *** [CMakeFiles/Makefile2:325: demo/CMakeFiles/aps-demo.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
fduncanh commented 8 months ago

This entry in src/CMakeLists.txt should ensure that libaps.so gets the third-party libs.

target_link_libraries(${PROJECT_NAME} PUBLIC                                                                                                                                                                                                   
    curve25519                                                                                                                                                                                                                                 
    ed25519                                                                                                                                                                                                                                    
    hlsparser                                                                                                                                                                                                                                  
    opensslaes                                                                                                                                                                                                                                 
    playfair                                                                                                                                                                                                                                   
)
thiccaxe commented 8 months ago

I set it to private, then explcitly imported all third-party files into demo. Doesn't work.

fduncanh commented 8 months ago

There are 5 static libs build in third-party why is only libplayfair.a a problem?

 curve25519                                                                                                                                                                                                                                   
  ed25519                                                                                                                                                                                                                                      
  hlsparser                                                                                                                                                                                                                                    
  opensslaes                                                                                                                                                                                                                                   
  playfair
thiccaxe commented 8 months ago

libplayfair

maybe libplayfair is just the first one that runs into an error?

thiccaxe commented 8 months ago

ok, my theory is that we need to wrap everything with ifdef cpp

fduncanh commented 8 months ago

I commented out the call in src/ap_crypto.cpp to playfair_decrypt, and the demo built. which suggests it is specific to libplayfair.a

thiccaxe commented 8 months ago

IT BUILT

thiccaxe commented 8 months ago

IT WORKS

fduncanh commented 8 months ago

OK a C to cpp issue then. Yes the new version with a dummy has a extern C wrapper.....

thiccaxe commented 8 months ago

audio segfaults, but streaming videosharing works.

fduncanh commented 8 months ago

what is the specific fix?

thiccaxe commented 8 months ago

check https://github.com/thiccaxe/apsdk-public/tree/streaming

fduncanh commented 8 months ago

@thiccaxe I consolidated things cleanly in

https://github.com/FDH2/apsdk-public/tree/older

using your approach of putting the fp_response and fp_header into playfair.c. So just add playfair to apsdk-code without otherwise changing it.

I should have looked at the .h files for the other third-party C codes: they all have the cpp guard.

output is interesting.

  1. for current UxPlay, there is info on X-Apple-Session-ID which dsafa22 just said to ignore.

  2. For YouTube video, FCUP etc. it sends a huge plist but we dont have something that handles it, and get DEBUG]Action type: unhandledURLResponse

  3. I get audio in mirror mode, but not on alac mode.

fduncanh commented 8 months ago

In apsdk, the Client pair-setup call occurs after fpsetup, and contains X-Apple-Session-ID in its header, which doesnt happen in UxPlay. Maybe because the "features" code is different? (I am using unmodified apsdk features).

fduncanh commented 8 months ago

The alac mode failure is that only aac audio seems to be supported ? [libfdk_aac @ 0x7ff09831f140] aacDecoder_DecodeFrame() failed: 400c

EDIT: yes there is no ALAC decoder linked to apsdk. There is some provision for ALAC in the code, though.

fduncanh commented 8 months ago

These are the requests (RTSP and HTTP) that apsdk responds to.

UxPlay only handles RTSP (RAOP) requests, probably only gets RTSP because of features bit settings?

void ap_airplay_connection::initialize_request_handlers() {
  // The request route table
  request_route_t routes_table[] = {
      {"RTSP", "OPTIONS", "", RH(options_handler)},
      {"RTSP", "POST", "/pair-setup", RH(post_pair_setup_handler)},
      {"RTSP", "POST", "/pair-verify", RH(post_pair_verify_handler)},
      {"RTSP", "POST", "/fp-setup", RH(post_fp_setup_handler)},
      {"RTSP", "SETUP", "*", RH(setup_handler)},
      {"RTSP", "GET", "/info", RH(get_info_handler)},
      {"RTSP", "POST", "/feedback", RH(post_feedback_handler)},
      {"RTSP", "RECORD", "*", RH(record_handler)},
      {"RTSP", "GET_PARAMETER", "*", RH(get_parameter_handler)},
      {"RTSP", "SET_PARAMETER", "*", RH(set_parameter_handler)},
      {"RTSP", "TEARDOWN", "*", RH(teardown_handler)},
      {"RTSP", "FLUSH", "*", RH(flush_handler)},
      {"RTSP", "POST", "/audioMode", RH(post_audioMode)},
      {"HTTP", "GET", "/server-info", RH(get_server_info_handler)},
      {"HTTP", "POST", "/fp-setup", RH(post_fp_setup_handler)},
      {"HTTP", "POST", "/fp-setup2", RH(post_fp_setup2_handler)},
      {"HTTP", "POST", "/reverse", RH(post_reverse_handler)},
      {"HTTP", "POST", "/play", RH(post_play_handler)},
      {"HTTP", "POST", "/scrub", RH(post_scrub_handler)},
      {"HTTP", "POST", "/rate", RH(post_rate_handler)},
      {"HTTP", "POST", "/stop", RH(post_stop_handler)},
      {"HTTP", "POST", "/action", RH(post_action_handler)},
      {"HTTP", "GET", "/playback-info", RH(get_playback_info_handler)},
      {"HTTP", "PUT", "/setProperty", RH(put_setProperty_handler)},
      {"HTTP", "POST", "/getProperty", RH(post_getProperty_handler)},
  };
[](url)
  // Register all the request handlers
  for (const auto& route : routes_table) {
    register_request_route(route);
  }
}

HTTP POST /action is involved in video streaming

UxPlay only handles RTSP requests. It would need to be expanded to handle the HTTP requests to do video streaming.

   if (!strcmp(method, "GET") && !strcmp(url, "/info")) {
        handler = &raop_handler_info;    *response = http_response_init("RTSP/1.0", 200, "OK");

    http_response_add_header(*response, "CSeq", cseq);
    //http_response_add_header(*response, "Apple-Jack-Status", "connected; type=analog");
    http_response_add_header(*response, "Server", "AirTunes/"GLOBAL_VERSION);

    logger_log(conn->raop->logger, LOGGER_DEBUG, "Handling request %s with URL %s", method, url);
    raop_handler_t handler = NULL;
    if (!strcmp(method, "GET") && !strcmp(url, "/info")) {
    } else if (!strcmp(method, "POST") && !strcmp(url, "/pair-pin-start")) {
        handler = &raop_handler_pairpinstart;
    } else if (!strcmp(method, "POST") && !strcmp(url, "/pair-setup-pin")) {
        handler = &raop_handler_pairsetup_pin;
    } else if (!strcmp(method, "POST") && !strcmp(url, "/pair-setup")) {
        handler = &raop_handler_pairsetup;
    } else if (!strcmp(method, "POST") && !strcmp(url, "/pair-verify")) {
        handler = &raop_handler_pairverify;
    } else if (!strcmp(method, "POST") && !strcmp(url, "/fp-setup")) {
        handler = &raop_handler_fpsetup;
    } else if (!strcmp(method, "OPTIONS")) {
        handler = &raop_handler_options;
    } else if (!strcmp(method, "SETUP")) {
        handler = &raop_handler_setup;
    } else if (!strcmp(method, "GET_PARAMETER")) {
        handler = &raop_handler_get_parameter;
    } else if (!strcmp(method, "SET_PARAMETER")) {
        handler = &raop_handler_set_parameter;
    } else if (!strcmp(method, "POST") && !strcmp(url, "/feedback")) {
        handler = &raop_handler_feedback;
    } else if (!strcmp(method, "RECORD")) {
        handler = &raop_handler_record;
    } else if (!strcmp(method, "FLUSH")) {
        handler = &raop_handler_flush;
    } else if (!strcmp(method, "TEARDOWN")) {
        handler = &raop_handler_teardown;
    }
fduncanh commented 8 months ago

The code for handling FCUP requests may have been removed (commented out) here

https://github.com/FDH2/apsdk-public/commit/597d6edf9f771da0c820807d24b4e9ababe6e310

EDIT: useful info: https://air-display.github.io/airplay-internal/media_cast_service.html

thiccaxe commented 8 months ago

https://openairplay.github.io/airplay-spec/video/http_requests.html this has most of the http requests documented as well. However, not all of them.

thiccaxe commented 8 months ago

I mentioned this a year ago, not sure if it is still true

it seems apple has updated their alac protocol, or is atleast using new modes which the alac decoder used does not support

thiccaxe commented 8 months ago

Some resources https://github.com/yuanhotel/airplay4j/blob/04f0d826562a69f541b77a107e6fa8e5604879fc/src/main/java/com/yutel/silver/util/AirplayUtil.java#L118 http://nto.github.io/AirPlay.html#video-httprequests https://github.com/jiaoyang623/screencast/blob/master/airplay/Bravo/assets/playback-info.plist https://github.com/ConnectSDK/Connect-SDK-Android/issues/226

-> very helpful https://github.com/monxarat/AirPlay-Receiver-on-Android

I just did some testing, and it looks casting appletv+ content to an appletv3 switches to an encrypted http protocol after some fp-setup calls, whereas ap-sdk does not.

thiccaxe commented 8 months ago

Additionally, youtube doesn't seem to work with a real apple tv—I don't know if I will be able to reverse-engineering it just from packet inspection

fduncanh commented 8 months ago

appleTV+ is encrypted. I have youtube app playing on atv3 right now It works. (also have atv 4k)

I'm not sure how to listen in with wireshark

thiccaxe commented 8 months ago

Maybe I should try updating the YouTube app and Apple TV.

To capture packets https://wiki.wireshark.org/HowToDecrypt802.11

thiccaxe commented 8 months ago

Put your wife card in monitor mode with airmon-ng

thiccaxe commented 8 months ago

It seems that youtube too is now most likely using some sort of an encrypted protocol.

GenghisKhanDrip commented 6 months ago

Were you able to get the demo to build and run? I tried cloning the FDH2 clone (older branch) and couldn’t manage to build it on Linux, any instructions would be appreciated!