Trying to add h265 support (for modern clients with wired connection) (was *** ERROR: unsupported h265 video detected)

DYY-Studio commented 8 months ago

iOS 16.5.1 iPad Pro 4 (11-inch)

s 1920x1342@60
nohold
FPSdata
fps 60

When I use Ethernet to AirPlay（Both iPad and my Windows Computer）, it shows the error message, but Wi-Fi would not. Do we have any plan to support the h265 video？Thanks.

By the way, Wi-Fi here is too bad to AirPlay a fluent video, that's why I want to use Ethernet.

thiccaxe commented 8 months ago

Take a look at the discussion in #231

DYY-Studio commented 8 months ago

Thanks, I'll wait for good news.

Should I close the issue?

fduncanh commented 8 months ago

The h265 seems to be decoded properly, and now its recognized, it should be easy to handle =.

But it needs a gstreamer pipeline to be written. As of now I don't know if it will just be modifying "h264" to "h265", or more complicated. I will need to get a macO or iPadOS client to deliver h265. It seems ethernet connections instead of wifi are what triggers it?

@DYY-Studio, how to you get the iPad to use ethernet? Is there a dongle for that? I have one for a mac M2 with usb c and an iPad m2 with usb c as well.

DYY-Studio commented 8 months ago

@fduncanh Yes. I use an UGREEN USB-C Hub (Type-C to 2*USB-A + USB-C + HDMI + 3.5mm) and then plug an UGREEN 1Gbps USB-A to RJ45 (AX88179A) into one of the USB-As.

The H265 just happened when I use Ethernet on iPad. I have tried using Ethernet on my Windows laptop but Wi-Fi on my iPad, it didn't show the error.

I haven't tried whether the connection between my laptop and iPad can reach full 1Gbps, but the Speed Test on both devices show that the connections to Internet are 1Gbps.

thiccaxe commented 8 months ago

I tried with a usb-c to ethernet adapter. I was achieving full gigabit speeds on LAN, but the ipad decided to use h264. I think I have an older model so I probably won't be able to help out here

fduncanh commented 8 months ago

@DYY-Studio

I have a " USB C to Ethernet Adapter, uni Driver Free RJ45 to USB C [Thunderbolt 3/4 Compatible], 1Gbps Type-C Gigabit Ethernet LAN Network Adapter for MacBook Pro"

What else should I need? (M2 macbook pro and M2 ipad pro are available as test clients)

?just plug into client usb port and connect to 1GB hub of local network?

DYY-Studio commented 8 months ago

@fduncanh I think that's all. I just do like this.

The cable go directly to the router's 1Gbps port, and my laptop was using the router's Wi-Fi.

It seems that it's just about iPad's Ethernet connection.

Both Wi-Fi

H264

Both Ethernet

H265

iPad Ethernet, Laptop Wi-Fi

H265

Laptop Ethernet, iPad Wi-Fi

H264

All the Ethernet connection I used is 1Gbps.

fduncanh commented 8 months ago

@ionull @DYY-Studio We have confirmed and duplicated the issue. We will see what can be done.

fduncanh commented 8 months ago

The difficulty in getting h265 supported is that the first video packet (which is unencrypted, "type 0x01") carries no payload:

in h264 video, it delivers the SPS (sequence parameter set NAL unit) and PPS (picture parameter set NAL unit) , needed for the decoding process)
in h265 video, there should be 3 such NAL units, SPS, PPS and VPS (Video parameter set), but the unencrypted initial packet does not seem to contain them. The first encrypted packet just contains the expected initial IDR video slice, and the subsequent packets contain regular video slices. But without the VPS+SPS+PPS, feeding them to the gstreamer h265parse element produces a "malformed NAL error".

??? why is the type 0x01 packet (see lib/raop_rtp_mirror.c) missing the VPS+SPS+PPS data? Where to find it?


Client identified as User-Agent: AirPlay/710.79.1
Accepted IPv4 client on socket 38
Local: 10.5.5.1
Remote: 10.5.5.2
raop_rtp_mirror starting mirroring
received video packet, header                   (header of initial unencrypted packet)
00 00 00 00 01 00 16 01 b8 c6 67 f5 83 0e 00 00    <===  the first 4 bytes 00 00 00 00 are the (zero)
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00          payload length, next byte is type 0x01 (unencrypted)
00 00 00 00 00 00 00 00 00 00 20 45 00 00 b4 44   <===  this data is picture width and height
00 00 00 00 00 00 00 00 00 00 20 45 00 00 b4 44   <===  this is width_source and height_source
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

Received unencrypted codec packet from client: payload_size 0 header 01 00 16 01  ts_client = 3715.958615
raop_rtp_mirror, discard type 0x01 packet with no payload

received video packet, header            this is header of first encrypted packet
cb fa 03 00 00 10 00 00 b8 c6 67 f5 83 0e 00 00      <== first 4 bytes are payload  length (littleendian)    
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00     "type = 0x00"  (next byte)
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

decrypted payload,   size 260811    (size adds the initial 4 bytes 00 00 00 01 that code for a new NAL)
00 03 fa c7 28 01 af 56 0b 47 0d fa 07 f8 32 a1     <=== first 4 bytes repeat length  (bigendian), next
11 03 ac 16 b9 05 43 ef 8c 84 6d aa a2 6d 61 3d       two bytes 28 01 are the h265 header, indicating
23 03 dc 0a 10 d7 46 39 05 de 82 f8 ab df 3b a8       that it is a type 20 IDR video slice NAL unit.
d6 a7 78 2f ce b2 d9 3a d3 eb 90 a6 58 9a 9a a5 
c9 f6 aa d8 a0 fa 50 a9 45 a3 b8 88 aa 02 c7 3e 
1d 2a 81 9a 1b 48 cf c0 26 a7 a6 14 a3 41 cb a7 
6d 52 97 f3 7f a3 e6 a1 96 47 67 fa d9 fb eb f3 
<snip>

https://www.codeproject.com/tips/896030/the-structure-of-hevc-video

EDIT: maybe the VPS+SPS+PPS data is in the IDR slice but not as separated NALS?? If so, how to tell gstreamer h265parse ?

fduncanh commented 8 months ago

*Thinking some more about this, I'm wondering whether there is in fact a payload with the unencrypted packet containing the VPS, SPS and PPS NAL units, but Apple unhelpfully doesn't give the payload size when it is unencrypted. The test would be to see if more data can be read from the socket after getting the header. Since the first four bytes of the NAL units contain their length (which we then replace with the NAL start code 00 00 00 01) this would be a viable way to send the data.

Encrypted packets need to have a known size before decryption, and since their payload is encrypted, the size can't be read from the first 4 payload bytes, so Apple has to specify them in the packet header, but could get away with not specifying them when the payload is unencrypted (just to be mean!)

I will test this idea when I have the time.

This didn't work. When I read from the socket after the initial type 0x01 packet with no payload, I just get the first type 0x00 packet with an encrypted payload.
so how does Apple deliver the three NAL units VPS, SPS, PPS needed to set up decoding the h265 data???
without solving this puzzle, adding support for h265/hevc video "hits a brick wall"

EDIT: It might be that the new h265 protocol won't send any unencrypted data to an AirPlay Server that does not use the newer AirPlay 2 protocol that used fully-encrypted server-client communications.??? Some wireshark studies on how h265 works on an AppleTV 4K might help, but its not clear if any other project besides UxPlay is interested in AirPlay video as opposed to Audio. Suggestions welcomed!

fduncanh commented 8 months ago

@DYY-Studio

You write

When I use Ethernet to AirPlay（Both iPad and my Windows Computer）

Just out of interest:

Are you using AirMyPC on the Windows computer (or something else)?

DYY-Studio commented 8 months ago

@fduncanh No, I use UxPlay only now.

Other softwares I have tried once but in that time I didn't have the idea that to use Ethernet to enhance stability, and also I didn't have iPad.

Some of the software I tried limit the bitrate of AirPlay, the using time or the display size. That makes me try to find a better one, which leads me to UxPlay. I remember I have tried AirMyPC once, but it seems like I didn't understand how to run it on a Windows PC that time?

I can't remember what softwares I have tried, sorry.

fduncanh commented 8 months ago

@DYY-Studio

I see. Am I correct that you are running UxPlay as an AirPlay server on a windows PC, using it to mirror an iPad client?

AirMyPC emulates an AirPlay client on windows

DYY-Studio commented 8 months ago

@fduncanh Yes. After all I don't have Apple TV and Mac. And generally I prefer to use Microsoft Remote Desktop.

I want to replace iOS system screen-recorder but a good HDMI video capture card is too expensive. AirPlay seems like a good choice for its high bitrates and full resolution display. (System record only provide 1920x1342, with long saving time to scale.)

fduncanh commented 7 months ago

This is an issue that can't be solved without understanding the Apple AirPlay2 protocol better. A useful "fix" would be to systematically see if any of the undocumented bits in the two settings bytes (se lib/dnssdint.h) perhaps tell the client NOT to use h265/hevc coding when connected using wired ethernet

closing for now.

shuax commented 1 month ago

I tried to copy the file header vps+sps+pps from other h265 streams. Obviously I failed. I continued to try more possibilities.

fduncanh commented 1 month ago

The mystery is WHERE does Apple send the vps+sps+pps?, or what causes the client to NOT send them to UxPlay with the unencrypted packet

thiccaxe commented 1 month ago

If nothing else I can suggest that you plug in your phone into your laptop and use idevice tools' logger to see if there are any clues

FDH2 / UxPlay

Trying to add h265 support (for modern clients with wired connection) (was *** ERROR: unsupported h265 video detected) #233