search

FDH2 / UxPlay

AirPlay Unix mirroring server
GNU General Public License v3.0
1.34k stars 72 forks source link

(unsupported h265/hevc video)Larger resolution than 1920x1080 (h265 video from recent-generation client with wired connection, not wifi) #231

Closed ionull closed 8 months ago

ionull commented 8 months ago

When I set it to 2560x1440 or larger, there's only error log from VLC: unexpected partitioned VCL NAL unit: nalu_type = 2, ref_idc = 0, nalu_size = 3793,processed bytes 4, payloadsize = 3797 nalus_count = 1 unexpected partitioned VCL NAL unit: nalu_type = 2, ref_idc = 0, nalu_size = 3793,processed bytes 4, payloadsize = 3797 nalus_count = 1

fduncanh commented 8 months ago

Thanks for the report.

What is the iOS device? (is it one of the larger new ipads?)

Does the image get displayed at all? Or are you just reporting the error message?

ionull commented 8 months ago

Hi, it's mirroring from my MacBook Pro to my iMac 5k with Ubuntu. The speed should be ok because they are linked by cable. It works well when iMac running on macOS. There is no any image displayed when set larger resolution than 1080p.

Sent from Gmail Mobile

On Sat, 21 Oct 2023 at 10:15, fduncanh @.***> wrote:

Thanks for the report.

What is the iOS device? (is it one of the larger new ipads?)

Does the image get displayed at all? Or are you just reporting the error message?

— Reply to this email directly, view it on GitHub https://github.com/FDH2/UxPlay/issues/231#issuecomment-1773608147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB3E57X5ZZHAIPZ2IRMLULYAMV37AVCNFSM6AAAAAA6GZHULWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZTGYYDQMJUG4 . You are receiving this because you authored the thread.Message ID: @.***>

fduncanh commented 8 months ago

looking at the code, it seems this is just an informational message about the type of video data package ("NAL unit") received. We havent seen "slice data" NAL's before.
Do they fail to get processd properly?

If this can be reproduced it can probably get fixed.

                    int nalu_type = payload_decrypted[nalu_size] & 0x1f;
                    int ref_idc = (payload_decrypted[nalu_size] >> 5);          
                    switch (nalu_type) {
                    case 14:  /* Prefix NALu , seen before all VCL Nalu's in AirMyPc */
                    case 5:   /*IDR, slice_layer_without_partitioning */
                    case 1:   /*non-IDR, slice_layer_without_partitioning */
                        break;
                case 2:   /* slice data partition A */
                    case 3:   /* slice data partition B */
                    case 4:   /* slice data partition C */
                        logger_log(raop_rtp_mirror->logger, LOGGER_INFO,
                                   "unexpected partitioned VCL NAL unit: nalu_type = %d, ref_idc = %d, nalu_size = %d,"
                                   "processed bytes %d, payloadsize = %d nalus_count = %d",
                                   nalu_type, ref_idc, nc_len, nalu_size, payload_size, nalus_count);
                        break;
                    case 6:
                        if (logger_debug) {
                            char *str = utils_data_to_string(payload_decrypted + nalu_size, nc_len, 16); 
                            logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG, "raop_rtp_mirror SEI NAL size = %d", nc_len);     
                            logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG,
                                       "raop_rtp_mirror h264 Supplemental Enhancement Information:\n%s", str);
                            free(str);
                        }
                        break;
                    case 7:
                        if (logger_debug) {
                            char *str = utils_data_to_string(payload_decrypted + nalu_size, nc_len, 16); 
                            logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG, "raop_rtp_mirror SPS NAL size = %d", nc_len);     
                            logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG,
                                       "raop_rtp_mirror h264 Sequence Parameter Set:\n%s", str);
                            free(str);
                        }
                        break;
                    case 8:
                        if (logger_debug) {
                            char *str = utils_data_to_string(payload_decrypted + nalu_size, nc_len, 16); 
                            logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG, "raop_rtp_mirror PPS NAL size = %d", nc_len);     
                            logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG,
                                       "raop_rtp_mirror h264 Picture Parameter Set :\n%s", str);
                            free(str);
                        }
                        break;
                    default:
                        logger_log(raop_rtp_mirror->logger, LOGGER_INFO,
                                   "unexpected non-VCL NAL unit: nalu_type = %d, ref_idc = %d, nalu_size = %d,"
                                   "processed bytes %d, payloadsize = %d nalus_count = %d",
                                   nalu_type, ref_idc, nc_len, nalu_size, payload_size, nalus_count);
            break;
            }
                    nalu_size += nc_len;
                }
ionull commented 8 months ago

It can be reproduced every time. I am not familiar with C programming but can help testing.

Sent from Gmail Mobile

On Sat, 21 Oct 2023 at 10:28, fduncanh @.***> wrote:

looking at the code, it seems this is just an informational message about the type of video data package ("NAL unit") received. We havent seen "slice data" NAL's before. Do they fail to get processd properly?

If this can be reproduced it can probably get fixed.

                int nalu_type = payload_decrypted[nalu_size] & 0x1f;
                int ref_idc = (payload_decrypted[nalu_size] >> 5);        
                switch (nalu_type) {
                case 14:  /* Prefix NALu , seen before all VCL Nalu's in AirMyPc */
                case 5:   /*IDR, slice_layer_without_partitioning */
                case 1:   /*non-IDR, slice_layer_without_partitioning */
                    break;
          case 2:   /* slice data partition A */
                case 3:   /* slice data partition B */
                case 4:   /* slice data partition C */
                    logger_log(raop_rtp_mirror->logger, LOGGER_INFO,
                               "unexpected partitioned VCL NAL unit: nalu_type = %d, ref_idc = %d, nalu_size = %d,"
                               "processed bytes %d, payloadsize = %d nalus_count = %d",
                               nalu_type, ref_idc, nc_len, nalu_size, payload_size, nalus_count);
                    break;
                case 6:
                    if (logger_debug) {
                        char *str = utils_data_to_string(payload_decrypted + nalu_size, nc_len, 16);
                        logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG, "raop_rtp_mirror SEI NAL size = %d", nc_len);       
                        logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG,
                                   "raop_rtp_mirror h264 Supplemental Enhancement Information:\n%s", str);
                        free(str);
                    }
                    break;
                case 7:
                    if (logger_debug) {
                        char *str = utils_data_to_string(payload_decrypted + nalu_size, nc_len, 16);
                        logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG, "raop_rtp_mirror SPS NAL size = %d", nc_len);       
                        logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG,
                                   "raop_rtp_mirror h264 Sequence Parameter Set:\n%s", str);
                        free(str);
                    }
                    break;
                case 8:
                    if (logger_debug) {
                        char *str = utils_data_to_string(payload_decrypted + nalu_size, nc_len, 16);
                        logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG, "raop_rtp_mirror PPS NAL size = %d", nc_len);       
                        logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG,
                                   "raop_rtp_mirror h264 Picture Parameter Set :\n%s", str);
                        free(str);
                    }
                    break;
                default:
                    logger_log(raop_rtp_mirror->logger, LOGGER_INFO,
                               "unexpected non-VCL NAL unit: nalu_type = %d, ref_idc = %d, nalu_size = %d,"
                               "processed bytes %d, payloadsize = %d nalus_count = %d",
                               nalu_type, ref_idc, nc_len, nalu_size, payload_size, nalus_count);
      break;
      }
                nalu_size += nc_len;
            }

— Reply to this email directly, view it on GitHub https://github.com/FDH2/UxPlay/issues/231#issuecomment-1773613431, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB3E564RUDWSB66OS4GFD3YAMXK7AVCNFSM6AAAAAA6GZHULWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZTGYYTGNBTGE . You are receiving this because you authored the thread.Message ID: @.***>

fduncanh commented 8 months ago

The message is just informational.

What actually happens with the video?

What kind of iOS client device is producing this video?

iPad/ iPhone, which model, which version of iOS / iPadOS etc.

what uxplay command is being used (what options?)

fduncanh commented 8 months ago

NAL units are part of H264 video format data

https://en.wikipedia.org/wiki/Network_Abstraction_Layer

EDIT: the issue turns out to be that h265 video is being sent by the client, and uxplay is trying to interpret h265 NAL units as if they were h264.

ionull commented 8 months ago

Hi There is no "iOS" device but a MacBook Pro using screen mirroring. Launch command: uxplay -vs waylandsink -fs Config: s 2560x1440

fduncanh commented 8 months ago

OK This can be tested (maybe only next week , or the week after).

Right now its assumed that each video NAL unit is a complete video frame, maybe the three "partitioned slices" should be grouped together and sent to gstreamer as a group.

ionull commented 8 months ago

Hi, thanks for response.

fduncanh commented 8 months ago

Its probably because iOS clients dont send "partitioned slices" so we did not see them before.

I'm guessing the larger image needed partitioning. If we can understand the H264 details, this can probably be fixed

https://stackoverflow.com/questions/19642736/count-frames-in-h-264-bitstream

fduncanh commented 8 months ago

Just a thought:

can you check if an older uxplay release (say 1.62) also has the issue?

The "video resumed" line in the debug output might come from a recent change

look in "releases" on the uxplay web page

fduncanh commented 8 months ago

For reference:

https://web.archive.org/web/20141016205250/http://www.szatmary.org/blog/25#more-25

ionull commented 8 months ago

Hi, I just make install 1.62. It can display but the resolution is the same, no 2560x1440

fduncanh commented 8 months ago

Thanks.

The issue will be looked at, my guess is it can be fixed, after finding out how to process NAL units with "partitioned slices"

ionull commented 8 months ago

BTW, when I set fps to 60(no resolution set), there's the same error log without any image displayed.

fduncanh commented 8 months ago

the fps setting tell the client what the maximum acceptable fps is. Its up to the client to decide what fps to use.

No one is available to investigate this issue at the moment. In a week's time probably.

ionull commented 8 months ago

No hurry, thanks for your hard working ;)

fduncanh commented 8 months ago

finally had the opportunity to test.

uxplay -s 2650x1440

seems to be working fine to mirror a macbook M2 2023 running macOS 13.6 ventura. The debug output shows that the client (the macbook) is indeed sending video with 2560x1440 The screen image filled the big screen on the uxplay server., which can display 2650x1440

raop_rtp_mirror accepting client

Received unencrypted codec packet from client: payload_size 39 header 01 00 16 01  ts_client = 155214.533833
raop_rtp_mirror: unidentified extra header data  0.000000, 0.000000
begin video stream wxh = 2560x1440; source 2560x1440
raop_rtp_mirror width_source = 2560.000000 height_source = 1440.000000 width = 2560.000000 height = 1440.000000
raop_rtp_mirror: SPS+PPS header size = 6
raop_rtp_mirror h264 SPS+PPS header:
01 64 00 33 ff e1 

raop_rtp_mirror SPS NAL size = 20
raop_rtp_mirror h264 Sequence Parameter Set:
27 64 00 33 ac 56 80 28 00 b5 a6 e0 20 20 20 da 
08 84 65 80 

raop_rtp_mirror PPS NAL size = 4
raop_rtp_mirror h264 Picture Parameter Set:
28 ee 37 27 

remainder size = 4
remainder of SPS+PPS packet:
fd f8 f8 00 

video renderer paused
raop_rtp video: now = 1698172794.698381, ntp = 1698172794.638531, latency = 0.059850, ts = 155214.533833, 00 10 00 00 
raop_rtp_mirror SEI NAL size = 30
raop_rtp_mirror h264 Supplemental Enhancement Information:
06 05 1a 47 56 4a dc 5c 4c 43 3f 94 ef c5 11 3c 
d1 43 a8 01 ff cc cc ff 02 00 7a 12 00 80 

video renderer resumed
Begin streaming to GStreamer video pipeline
Looking for X11 UxPlay Window, attempt 1
raop_rtp video: now = 1698172794.708889, ntp = 1698172794.671864, latency = 0.037025, ts = 155214.567167, 00 00 00 00 
video renderer resumed
Looking for X11 UxPlay Window, attempt 2
httpd receiving on socket 24

For comparison, here below is your debug output

clearly the PPS NAL size is wrong ("101820") the PPS NAL unit is very short. That 101820 byte packet is the video data that follows the PPS (picture parameter set) NAL


raop_rtp_mirror accepting client

Received unencrypted codec packet from client: payload_size 0 header 01 00 16 01  ts_client = 337090.007167
raop_rtp_mirror, discard type 0x01 packet with no payload
raop_rtp video: now = 1697857792.219516, ntp = 1697857792.209996, latency = 0.009521, ts = 337090.007167, 00 10 00 00 
raop_rtp_mirror PPS NAL size = 101820
raop_rtp_mirror h264 Picture Parameter Set :
28 01 af 13 60 88 73 3e 2b 08 22 1e 48 9a 32 e5 
aa b2 42 6d 1b 19 e8 24 a7 94 1c 88 1e a1 41 09 
b0 77 50 c7 43 d1 0f b2 71 59 79 c3 15 93 92 34 
48 0a 20 14 a4 76 04 98 17 80 3a 00 30 03 36 ae 
90 8b 2a fa bf fd e0 d0 34 e4 2d 5e dc b4 9d 30 
e1 b0 24 86 cd 79 cc 30 e9 89 45 de 87 39 3b 72 
0c f1 14 6f b2 6c d2 d4 f2 21 a4 b0 26 60 a8 fe 
5c 82 ea 9a f9 36 0d 2c 62 bb cf f1 f4 e8 f9 70 
15 b4 a3 8a 69 7f 7f 8a 91 29 2f 0c de c8 10 f6 
40 c1 a7 0d 91 66 64 40 3e 8a 1b 42 74 f9 d8 98 
7c 6b 35 22 7b fa 49 67 19 ba 24 4d 63 06 93 f1 
0d f7 c0 cd c0 8e 90 73 b4 54 e7 fd 0b 64 ef 3d 
a5 f7 7d 30 d9 fe 79 4f a3 c7 de a1 fd 04 86 e2 
49 34 33 ef e9 16 a1 8f 7b 17 bc 14 ad 16 f3 d9 
7c b0 15 58 88 0c 91 6c 2e 38 02 15 8e 07 7b c9 
20 3e a6 81 0f ec d9 98 10 6c 95 2d 03 61 71 65 
97 06 2c 14 44 03 a7 af 50 23 93 1d b2 c3 69 7f 
4b bf b4 64 d3 7f c8 87 1b 6e 7d 90 a1 e7 dc a9 
76 f2 ca 76 8d cb 2c be cc 6f b2 6a 6b 76 16 be 
25 dd 15 62 1a 10 9b c3 df e7 96 44 03 f7 f6 77 
20 4e d0 c4 69 c1 53 1c 40 4b d5 76 6f df 29 da 
3d b6 0b b3 02 05 94 60 d7 80 15 7b 5d 4c 27 bf 
34 71 71 59 1f 5a b1 40 b1 2e c3 5f dd 7f d5 73 
9c e6 f4 ab ae d1 b0 35 78 97 24 38 65 b7 06 b1 
b0 e9 90 88 be 7a 2b 4b 9f 74 1b ba a9 55 b6 d5 
33 02 e8 88 bb 94 74 b5 69 d3 dc 53 f6 ea 70 d7 
1b 1b ff b7 74 04 6e 94 01 6f a2 76 95 f1 a0 93 
dc 83 3e 40 b2 ad 4c ca 1a 39 09 fa 78 bf 76 74 
5d ec dd 2d f7 f0 3a ca 9e 16 a5 eb 42 ba d1 e3 
c6 36 08 1a 53 8a 90 08 ad 46 fd f3 28 58 e6 fd 
54 81 3f 70 a7 5e 2c 6f 90 51 8b 60 b7 e3 e1 49 
83 54 ab 1b fc a3 93 e2 41 14 39 45 d9 bd dc bf 
f0 e6 77 b4 8a 10 f4 85 75 2f d2 1c ab 28 32 70 
6d 90 e8 6f 93 97 15 84 41 e1 f4 70 a1 63 18 13 
3d 9a 4b 2e e0 57 2f 69 ce 0f a1 f1 ce 06 57 19 
9c 74 b1 43 4e dd 14 ff 4b b1 b1 93 fb 56 0c 97 
27 58 d8 59 9e a4 fd c6 d4 b5 65 07 e5 59 af 2d 
37 18 17 83 2b b6 6e 81 f7 71 71 6c e6 82 88 76 
fa fb 65 4e 75 77 b1 22 54 38 0c 0c 1a 4d 03 0f 
d4 aa e8 ca 0a ed 3e ba cb b7 a8 38 97 01 e0 53 
aa 00 82 73 89 03 a3 87 7e ff 6e aa ed eb 7b 4f 
5c 81 f2 51 d2 2f 6e 53 cf 93 93 d2 bc 85 7e fe 
27 9d 6e b3 dc 9a bb 6d d8 8b 9e bf 17 c4 c0 18 
ef 2a 40 18 50 44 d7 10 85 b8 ff be b1 9f 91 f2 
33 ae da fd e0 2b 5b be 7e 27 3e e0 b5 76 de 25 
a2 96 22 91 2f 44 ce d0 78 65 5e 11 23 bb 15 d5 
a4 fe 20 2a af 5d fd 7c b3 36 b5 11 16 bd db 72 
0c 85 35 a5 89 29 e5 ce e3 a2 a8 96 63 cb 75 b3 
03 07 79 e0 44 27 43 9b 09 a4 07 73 42 23 c6 a9 
cf c8 95 19 1b 82 46 18 a1 4e 9a b4 78 dd 59 9f 
f7 c7 94 56 75 30 95 3f 2c e6 72 c3 27 7f 24 d7 
c7 74 00 b9 f1 11 48 95 99 98 e1 1a 83 99 f8 df 
c7 7a 2c b7 6e 31 15 9e 35 82 cb 39 a0 62 3f c7 
c9 f8 65 58 db 70 2b 7b 11 92 18 24 ef 0c a6 d5 
8a d2 c8 8c 44 e9 a1 9e 19 49 a7 f0 3b f8 56 39 
95 ea 35 e7 0c 08 e8 97 77 b9 66 1e 4c 47 c1 a8 
41 e4 d6 32 2e c8 fb 4f 72 9e aa bb 7b 18 3b 05 
b7 de 23 43 13 97 4f f5 46 c3 3f 4b 61 57 05 14 
6e a3 48 43 40 78 1e 41 e3 32 aa f1 75 30 63 95 
76 48 d0 ff 62 12 22 50 8d 2a ea 89 61 a0 5f dc 
03 1a d9 79 ad 55 8a 17 87 e0 80 0d d3 d3 be 0b 
22 82 dd 3e dd 71 7e 26 88 5a 0a 47 15 87 4b ea 
85 50 2f 2a 00 c4 03 f2 04 23 09 c0 82 43 e2 ed 
07 7c f7 73 42 ca f9 b7 93 bf be 72 02 31 7a 31 
93 e1 46 d9 15 f4 80 35 85 e7 32 65 01 e5 19 21 
dd 0e 8d 5b 17 99 30 b7 3a 6e b8 60 5d aa b6 70 
7f a1 1e c8 49 98 f0 74 4d b7 28 81 9c 21 c9 81 
d3 89 b1 72 93 1d 99 aa 1a 51 b3 49 1b bc 6f 86 
14 8a 5f ce c1 66 64 17 13 e7 71 ff c2 2f 46 2f 
4a 3c 76 e8 a0 78 93 b2 47 81 98 5e 64 36 a4 9d 
0f cc 7a 2b ec a9 98 d1 f1 c5 be 58 19 5d 49 ca 
b0 f5 65 3d c0 7a 5a 28 21 71 1d 0f 24 11 61 64 
79 73 23 f2 cb 79 ca 89 b8 bc 6b a9 de 00 6d 36 
72 c9 bd c6 4b 47 3d 16 58 49 b8 5e b0 4d 00 9a 
a1 92 15 d0 0a 6e fa 72 df 96 07 d8 db 0c 99 f1 
eb d2 99 fb a9 11 ec 36 76 94 4d c7 06 0e 14 1e 
a0 e0 f6 a2 a9 1b b4 49 c2 ea b8 93 f5 2a f8 cd 
6d f9 4d 02 35 16 28 82 5f 3f 2d e0 18 dc b7 12 
ed 2f b0 4f 41 01 be 9c 70 1f fa 06 c9 3e 98 14 
ad 74 66 6c 73 c1 8a 9d 04 01 a6 90 5d c6 56 31 
5e de f1 e9 e2 89 0a 7a 4d d7 35 9f f1 e3 fd 42 
3b 6b 41 56 b2 e3 b0 c5 64 4e 66 4a 53 d7 66 e6 
5f 2b 9e c9 cd d1 00 9d b5 43 a
video renderer resumed
Begin streaming to GStreamer video pipeline
raop_rtp video: now = 1697857792.265764, ntp = 1697857792.243329, latency = 0.022435, ts = 337090.040500, 00 00 00 00 
unexpected partitioned VCL NAL unit: nalu_type = 2, ref_idc = 0, nalu_size = 40265,processed bytes 4, payloadsize = 40269 nalus_count = 1
video renderer resumed
raop_rtp video: now = 1697857792.265798, ntp = 1697857792.276662, latency = -0.010865, ts = 337090.073833, 00 00 00 00 
unexpected partitioned VCL NAL unit: nalu_type = 2, ref_idc = 0, nalu_size = 34060,processed bytes 4, payloadsize = 34064 nalus_count = 1
fduncanh commented 8 months ago

@ionull Our test was on a macbook running Ventura 13.6. no strange partitioned NAL units were seen

What OS is your macbook using?

EDIT: yes I see your MacBook Pro is on macOS 14.0

<key>osVersion</key>
    <string>14.0</string>
    <key>ekey</key>
ionull commented 8 months ago

@fduncanh Hi, I am using macOS Sonoma (m1 pro mackbook 14)

fduncanh commented 8 months ago

OK since macOS ventura works perfectly, will need to update out test machine to Sonoma to investigate your issue.

fduncanh commented 8 months ago

@ionull

see debug output below (from after video starts):

raop_rtp_mirror accepting client

Received unencrypted codec packet from client: payload_size 39 header 01 00 16 01  ts_client = 204.833833
raop_rtp_mirror: unidentified extra header data  0.000000, 0.000000
begin video stream wxh = 2560x1440; source 2560x1440
raop_rtp_mirror width_source = 2560.000000 height_source = 1440.000000 width = 2560.000000 height = 1440.000000
raop_rtp_mirror: SPS+PPS header size = 6
raop_rtp_mirror h264 SPS+PPS header:
01 64 00 33 ff e1 

raop_rtp_mirror SPS NAL size = 20
raop_rtp_mirror h264 Sequence Parameter Set:
27 64 00 33 ac 56 80 28 00 b5 a6 e0 20 20 20 da 
08 84 65 80 

raop_rtp_mirror PPS NAL size = 4
raop_rtp_mirror h264 Picture Parameter Set:
28 ee 37 27 

remainder size = 4
remainder of SPS+PPS packet:
fd f8 f8 00 

video renderer paused

raop_ntp send time type_t=82 packetlen = 32, now = 1698218649.026339
80 d2 00 07 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 e8 e3 41 19 06 be 27 ad 

raop_ntp receive time type_t=83 packetlen = 32, now = 1698218649.031012 t1 = 205.159171, t2 = 205.159207
80 d3 00 07 00 00 00 00 e8 e3 41 19 06 be 27 ad 
83 aa 7f 4d 28 bf 75 3d 83 aa 7f 4d 28 c1 cd 06 

raop_ntp sync correction = 0
raop_rtp video: now = 1698218649.044293, ntp = 1698218648.704330, latency = 0.339963, ts = 204.833833, 00 10 00 00 
raop_rtp_mirror SEI NAL size = 30
raop_rtp_mirror h264 Supplemental Enhancement Information:
06 05 1a 47 56 4a dc 5c 4c 43 3f 94 ef c5 11 3c 
d1 43 a8 01 ff cc cc ff 02 00 5b 8d 80 80 

video renderer resumed
Begin streaming to GStreamer video pipeline
Looking for X11 UxPlay Window, attempt 1
raop_rtp video: now = 1698218649.056915, ntp = 1698218648.770997, latency = 0.285919, ts = 204.900500, 00 00 00 00 
video renderer resumed
Looking for X11 UxPlay Window, attempt 2
raop_rtp video: now = 1698218649.071294, ntp = 1698218648.804330, latency = 0.266964, ts = 204.933833, 00 00 00 00 
video renderer resumed
Looking for X11 UxPlay Window, attempt 3

*** X11 Windows: Use key F11 or (left Alt)+Enter to toggle full-screen mode

raop_rtp video: now = 1698218649.118451, ntp = 1698218649.137663, latency = -0.019212, ts = 205.267167, 00 00 00 00 
raop_rtp video: now = 1698218649.118537, ntp = 1698218649.170997, latency = -0.052460, ts = 205.300500, 00 00 00 00 
raop_rtp video: now = 1698218649.128378, ntp = 1698218649.204330, latency = -0.075952, ts = 205.333833, 00 00 00 00 
raop_rtp video: now = 1698218649.155681, ntp = 1698218649.237663, latency = -0.081982, ts = 205.367167, 00 00 00 00 
raop_rtp video: now = 1698218649.183962, ntp = 1698218649.270997, latency = -0.087035, ts = 205.400500, 00 00 00 00 
raop_rtp video: now = 1698218649.210272, ntp = 1698218649.304330, latency = -0.094057, ts = 205.433833, 00 00 00 00 
raop_rtp video: now = 1698218649.239367, ntp = 1698218649.337663, latency = -0.098296, ts = 205.467167, 00 00 00 00 
raop_rtp video: now = 1698218649.346909, ntp = 1698218649.370996, latency = -0.024088, ts = 205.500500, 00 00 00 00 
raop_rtp video: now = 1698218649.347399, ntp = 1698218649.404330, latency = -0.056930, ts = 205.533833, 00 00 00 00 

Received video streaming performance info packet from client: payload_size 194 header 05 00 00 00  ts_raw = 0
raop_rtp video: now = 1698218649.347684, ntp = 1698218649.437663, latency = -0.089979, ts = 205.567167, 00 00 00 00 
raop_rtp video: now = 1698218649.369867, ntp = 1698218649.470997, latency = -0.101130, ts = 205.600500, 00 00 00 00
ionull commented 8 months ago

Is it because my Linux is a "iMac 2014"?

fduncanh commented 8 months ago

@ionull

The problem in your debug seems to be 

raop_rtp_mirror accepting client

Received unencrypted codec packet from client: payload_size 0 header 01 00 16 01  ts_client = 337090.007167
raop_rtp_mirror, discard type 0x01 packet with no payload

the SPS+PPS unencrypted data that setups up the video "had no payload"

fduncanh commented 8 months ago

Is it because my Linux is a "iMac 2014"?

It shouldn't matter, you are running Ubuntu on it. Which Ubuntu exactly?

ionull commented 8 months ago

Ubuntu 22.04

fduncanh commented 8 months ago

will retest on Ubuntu 22.04 (on an intel PC)

ionull commented 8 months ago

I tried to airplay my MacBookPro mid 2012 with Mojave system, it can airplay to my Linux with 5k resolution set. However I can't find a way to cable link them. The resolution is only 1080p.

fduncanh commented 8 months ago

no problem on Ubunt 22.04.1 on intel pc

raop_rtp_mirror accepting client

Received unencrypted codec packet from client: payload_size 39 header 01 00 16 01  ts_client = 976.833833
raop_rtp_mirror: unidentified extra header data  0.000000, 0.000000
begin video stream wxh = 2560x1440; source 2560x1440
raop_rtp_mirror width_source = 2560.000000 height_source = 1440.000000 width = 2560.000000 height = 1440.000000
raop_rtp_mirror: SPS+PPS header size = 6
raop_rtp_mirror h264 SPS+PPS header:
01 64 00 33 ff e1

your issue is in the apparently empty data packet set by the client, not anything on the server, it appears.

I look to see if a simple code change could provide more information on that.

fduncanh commented 8 months ago

@ionull I believe you said your system worked for "uxplay -s 1920x1080"

Please post the debug trace (just the bit starting with "raop_rtp_mirror accepting client") in that (working) case.

ionull commented 8 months ago

uxplay-log-work.txt

fduncanh commented 8 months ago

You 1920x1080 debug is OK

 raop_rtp_mirror accepting client

Received unencrypted codec packet from client: payload_size 39 header 01 00 16 01  ts_client = 109163.918393
raop_rtp_mirror: unidentified extra header data  0.000000, 0.000000
begin video stream wxh = 1920x1080; source 1920x1080
raop_rtp_mirror width_source = 1920.000000 height_source = 1080.000000 width = 1920.000000 height = 1080.000000
raop_rtp_mirror: SPS+PPS header size = 6
raop_rtp_mirror h264 SPS+PPS header:
01 64 00 2a ff e1

@ionull

add some code at line 364 of lib/raop_rtp_mirror.c (just above "switch (*packet[4])

            {
          char *str = utils_data_to_string(packet,128, 16);
          printf("received video packet with header:\n%s\n",str);
          free (str);
        }

this will print the 128-byte header of each video packet received

ionull commented 8 months ago

uxplay-log-video-packet-header.txt

fduncanh commented 8 months ago

OK your first video packet header says it is unexcrypted but had no data attached to it. Normally this should carry some data called SPS and PPS that tell the h264 decoder about the video format.

The second packet is initial video data ( a " type 5 IDR VCL NAL unit")

the subsequent packets are also video data "type 1(?) IDR VCL NAL units, possibly type 2)

mystery!

fduncanh commented 8 months ago 
Received unencrypted codec packet from client: payload_size 0 header 01 00 16 01  ts_client = 110640.739497
raop_rtp_mirror, discard type 0x01 packet with no payload

In the packet below, it should not issue the PPS NAL_SIZE = ... message. I'll check the uxplay code to see how that happens

received video packet with header:
02 ab 04 00 00 10 00 00 49 b5 4f bd 30 b0 01 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

raop_rtp video: now = 1698222658.928188, ntp = 1698222658.923944, latency = 0.004245, ts = 110640.739497, 00 10 00 00 
raop_rtp_mirror PPS NAL size = 305918
raop_rtp_mirror h264 Picture Parameter Set :
fduncanh commented 8 months ago

I think that the beginning of the data in the second video packet below ( 28 01 af 13 ..) is the PPS prepended to the "type 5 IDR VCL NAL unit".

question: what is your m1 macbook pro sending data this way, while our m2 mackbook pro doesnt do this ???

raop_rtp video: now = 1698222658.928188, ntp = 1698222658.923944, latency = 0.004245, ts = 110640.739497, 00 10 00 00 
raop_rtp_mirror PPS NAL size = 305918
raop_rtp_mirror h264 Picture Parameter Set :
28 01 af 13 60 88 78 a0 50 4b 40 eb c1 bf 09 3a 
1c 92 34 10 ee 4a 88 25 ba eb ef 98 3c b2 6b 6e 
bd 0d 02 0b ac 8f d8 b4 ee 38 94 4e 25 9c 43 ee 
8c f9 22 02 59 54 2e 28 0c 4f bc 1d 55 37 3a 6d 
58 f1 a1 da 28 1c 10 10 00 ac fc e8 93 65 00 d2 
03 07 34 9f 84 78 5b 70 06 c2 de 3e 99 73 5f 70 
33 6e b0 24 43 ed 89 15 f1 2b 0c 13 14 c4 9f 35 
f4 c6 85 ef cb 71 d1 7a 87 3f 40 cf f7 6a 95 84 
e3 ea ab a9 1a 7e a9 5f 4d c4 95 b4 f3 46 93 96 
73 38 a8 95 79 80 47 31 62 af a5 8a fe b1 72 7b 
83 9d 1f 8c fd 2f d0 c2 b0 95 a7 fb fb 69 33 c5
ionull commented 8 months ago

I think maybe because I like both device by a cable? With Dhcp server on Linux side.

Sent from Gmail Mobile

On Wed, 25 Oct 2023 at 17:25, fduncanh @.***> wrote:

I think that the beginning of the data in the second video packet below ( 28 01 af 13 ..) is the PPS prepended to the "type 5 IDR VCL NAL unit".

question: what is your m1 macbook pro sending data this way, while our m2 mackbook pro doesnt do this ???

raop_rtp video: now = 1698222658.928188, ntp = 1698222658.923944, latency = 0.004245, ts = 110640.739497, 00 10 00 00 raop_rtp_mirror PPS NAL size = 305918 raop_rtp_mirror h264 Picture Parameter Set : 28 01 af 13 60 88 78 a0 50 4b 40 eb c1 bf 09 3a 1c 92 34 10 ee 4a 88 25 ba eb ef 98 3c b2 6b 6e bd 0d 02 0b ac 8f d8 b4 ee 38 94 4e 25 9c 43 ee 8c f9 22 02 59 54 2e 28 0c 4f bc 1d 55 37 3a 6d 58 f1 a1 da 28 1c 10 10 00 ac fc e8 93 65 00 d2 03 07 34 9f 84 78 5b 70 06 c2 de 3e 99 73 5f 70 33 6e b0 24 43 ed 89 15 f1 2b 0c 13 14 c4 9f 35 f4 c6 85 ef cb 71 d1 7a 87 3f 40 cf f7 6a 95 84 e3 ea ab a9 1a 7e a9 5f 4d c4 95 b4 f3 46 93 96 73 38 a8 95 79 80 47 31 62 af a5 8a fe b1 72 7b 83 9d 1f 8c fd 2f d0 c2 b0 95 a7 fb fb 69 33 c5

— Reply to this email directly, view it on GitHub https://github.com/FDH2/UxPlay/issues/231#issuecomment-1778864756, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB3E54ARCILK7B5BTYRY5LYBDLJVAVCNFSM6AAAAAA6GZHULWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZYHA3DINZVGY . You are receiving this because you were mentioned.Message ID: @.***>

fduncanh commented 8 months ago

I dont think it is about cable etc.

The message about PPS NAL size could come from two places. can you modify line 480 of raop_rtp_mirror.c , and run with debug again so I can tell which place it came from:

                      logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG, "raop_rtp_mirror PPS NAL size = %d", nc_len);

change this to

                     logger_log(raop_rtp_mirror->logger, LOGGER_DEBUG, "type 8: raop_rtp_mirror PPS NAL size = %d", nc_len);
fduncanh commented 8 months ago

@ionull

do this: at line 446, just before "switch (nalu_type)", add

   printf("*** nalu_type = %d, nalus_count = %d nc_len = %d\n", nalu_type, nalus_count, nc_len);
ionull commented 8 months ago

Hi, it's weird. After I change the code and add the line. I can airplay 2560 now...

ionull commented 8 months ago

Oh sorry, it seems that my cable disconnected.. That must be the issue!

fduncanh commented 8 months ago

I will make a special version of uxplay for you in the "testing" branch (not yet created) of uxplay, to give the info I need to debug your issue. I'm guessing that PPS and SPS info is being added in front of the second video packet, not sent unencrypted with the first packet, so I need to look at it in detail.

I'll let you know here when its ready.

ionull commented 8 months ago

uxplay-log-nalu-type.txt

ionull commented 8 months ago

Now, I almost know how the issue happen. When I plugin cable between two devices, the connection speed is high enough to make macOS airplay higher resolution than through WiFi.

ionull commented 8 months ago

Without cable (ethernet line), I can airplay 2560 by wifi, but it's super laggy.

ionull commented 8 months ago

The original display of iMac 2014 is 5120x2880 which requires high speed cable(either thunderbolt or ethernet) But I can't get thunderbolt work on Linux. I am using ethernet cable.

fduncanh commented 8 months ago

Still, something is perhaps not right with processing the PPS NAL unit.

I will make a special testing version, but maybe not right now

ionull commented 8 months ago

Thanks for you hard work, I will wait your update.

ionull commented 8 months ago

You can do it like me, connect ethernet to both devices. If you have a dock, just ethernet cable is ok (on both side). If you have no dock, you can find some thunderbolt to ethernet adapter. After connect both devices, you can setup a dhcp on Linux side( make sure the ip showed can ping both sides).

fduncanh commented 8 months ago

@ionull can you run uxplay from the testing branch (without the -d option) and post some results (showing your original error)

That way I can see what exactly the mac client is sending

https://github.com/FDH2/UxPlay/tree/testing

if you use git,

git clone http://github.com/FDH2/UxPlay
cd UxPlay
git checkout testing
makedir build
cd build
cmake ..
make
./uxplay -s 2560x1440