search

FDH2 / UxPlay

AirPlay Unix mirroring server
GNU General Public License v3.0
1.35k stars 72 forks source link

Connection works with iPhone, but not with Macbook (M1, apple silicon, running Big Sur or Monterey macOS 11.x) (GStreamer problem with format of video stream from new Macbooks) #73

Closed amusingimpala75 closed 2 years ago

amusingimpala75 commented 2 years ago

When connecting with an iPhone 6, it connects fine. The screen shows up, and acts like it should. But when I connect with an M1 MacBook Air, the initial connection takes about 5 seconds before the console (for the UxPlay server) logs anything, and while the logging looks the same, no window pops up.

fduncanh commented 2 years ago

Haven't tested uxplay with m1 mac clients.

run uxplay -d while trying to connect, and compare your output to https://github.com/FDH2/UxPlay/files/8102988/uxplay_debug_2022-02-19.txt

report how far the connection process gets.

amusingimpala75 commented 2 years ago

Here, I did a diff of the expected log vs. what I got: uxplay_diff.log

By the way, the server is being run on a mid-2011 iMac running Debian 11.

amusingimpala75 commented 2 years ago

Here is the non diff-ed form, was just trying to be helpful. uxplay_nondiff.log

fduncanh commented 2 years ago

sorry, I realized afterwards I could just reverse diff.

raop_rtp_mirror video ntp = 1646610068820524, now = 1646610068839657, latency = 19133
raop_rtp_mirror video ntp = 1646610068853857, now = 1646610068883265, latency = 29408

Received video streaming performance info packet from client   ----------------------THIS HAPPENS ONCE PER SECOND.
raop_rtp_mirror video ntp = 1646610068887190, now = 1646610068908885, latency = 21695
raop_rtp_mirror video ntp = 1646610068920524, now = 1646610068939642, latency = 19118

httpd receiving on socket 30
conn_request

POST /feedback RTSP/1.0     <-- CLIENT IS ASKING IF SERVER IS STILL LISTENING   (every 2 secs)
CSeq: 12
DACP-ID: FA56E95AF5E51230
Active-Remote: 3236303819
User-Agent: AirPlay/600.8.1

Handling request POST with URL /feedback
raop_handler_feedback

RTSP/1.0 200 OK    <----- SERVER TELLS CLIENT, YES I'M STILL HERE
CSeq: 12 
Server: AirTunes/220.68 

(CLIENT SEEMS TO HAVE STOPPED STREAMING <  DOES IT HEAR THE SERVER'S ANSWER?)

Received video streaming performance info packet from client
raop_ntp send_len = 32, now = 1646610070431191        <---THIS HAPPENS EVERY 3secs
raop_ntp receive time type_t packetlen = 32
raop_ntp sync correction = -34535             <---------------THE SERVER GOT THE TIME SIGNAL FROM THE CLIENT OK.

Received video streaming performance info packet from client
httpd receiving on socket 30
conn_request

POST /feedback RTSP/1.0
CSeq: 13
DACP-ID: FA56E95AF5E51230
Active-Remote: 3236303819
User-Agent: AirPlay/600.8.1

Handling request POST with URL /feedback
raop_handler_feedback

RTSP/1.0 200 OK 
CSeq: 13 
Server: AirTunes/220.68 

Received video streaming performance info packet from client

Received video streaming performance info packet from client
raop_ntp send_len = 32, now = 1646610073456483
raop_ntp receive time type_t packetlen = 32
raop_ntp sync correction = 0
httpd receiving on socket 30
conn_request

POST /feedback RTSP/1.0
CSeq: 14
DACP-ID: FA56E95AF5E51230
Active-Remote: 3236303819
User-Agent: AirPlay/600.8.1

Handling request POST with URL /feedback
raop_handler_feedback

RTSP/1.0 200 OK 
CSeq: 14 
Server: AirTunes/220.68 

raop_rtp_mirror tcp socket closed
httpd receiving on socket 30

Somehow the client just stopped streaming, as if it thought the server was no longer there. I dont know what is going on here. sorry. I hope its not some protocol change in the new macs.

collect and post a few more (say 3) examples of debug logs, just the parts starting at

raop_rtp_mirror accepting client
begin video stream wxh = 1920x1080; source 1920x1080
raop_rtp_mirror width_source = 1920.000000 height_source = 1080.000000 width = 1920.000000 height = 1080.000000
raop_rtp_mirror sps size = 21
raop_rtp_mirror pps size = 4
raop_rtp_mirror video ntp = 1646610067937189, now = 1646610067958970, latency = 21781
Begin streaming to GStreamer video pipeline

until they end with 

raop_rtp_mirror tcp socket closed
httpd receiving on socket 30
conn_request

TEARDOWN rtsp://192.168.1.205/14019776013084737891 RTSP/1.0
fduncanh commented 2 years ago

@amusingimpala75 the error occurs at line 804 in https://github.com/FDH2/UxPlay/files/8194771/uxplay_nondiff.log

rtp_mirror tcp socket closed.

A bug involving that error was recently worked on .

Please try again with the latest UxPlay, again post a debug log.

There were two places that give the identical error message: now they are distinguishable,

amusingimpala75 commented 2 years ago

Here is another log beginning with the raop_rtp_mirror accepting client uxplay.log I will mention, I do disconnect from the server from the laptop. THe laptop recognizes the connection as being connected, but the server does not show any window, so I disconnect from the server before killing it.

fduncanh commented 2 years ago

How long are you waiting until you disconnect? Wait maybe one minute after connection happens.

I'm trying to understand whether the client or you is generating the TEARDOWN request.

Please use control-C in the terminal to terminate, not a command on the client (this can be seen in the debug log). and give me a new debug log

Booth1983 commented 2 years ago

I have the same issue with M1 Macbook Air with latest update. It says on console, that the connection is active, but no window is opened. I can even configure the usage of the Airplay-Screen on my Macbook.

Maybe I have time to run in debug mode later this week.

Ipad and iPhone work fine.

fduncanh commented 2 years ago

Its possible that Apple have done something with the protocol on new M1 Macs or on Monterrey in general.

Please do a debug run, and let it go on for some time after the window fails to open, and terminate it with control-c in the linux terminal, not with the mac.

fduncanh commented 2 years ago

@amusingimpala75 's debug log shows that there is no obvious protocol change, as the setup up, worked, and the video stream is streaming with correct decryption.

But I need to see what the mac is doing after streaming starts, and he confused things by stopping it streaming from the mac.

Just terminate on the linux terminal with ctrl=C. I minute of data at least after streaming starts would be good.

amusingimpala75 commented 2 years ago

@Booth1983 explains it much better than myself. What he describes is exacted what is happening to me.

Here's another log, terminated like you requested: uxplay.log

fduncanh commented 2 years ago

Looks like there is no problem with AIrPlay itself.....

Lets try to debug GStreamer (dont use -d option on UxPlay)

export GST_DEBUG=2 uxplay

what do you get with this?

fduncanh commented 2 years ago

Well, when I say "no problem with AIrplay", I mean with the protocol. You do seem to have what might be a network problem, gaps in the record when streaming stops for a while, then resumes

fduncanh commented 2 years ago

also try uxplay -FPSdata

the client (the mac) send data about its streaming rate once per second. Does this give any clues?

shuco85 commented 2 years ago

Dear FDH2 team, I have the same issue. UxPlay is working fine with iPhone but not with MacBook Pro M1.

The output's log is sending the following message in loop:

_0:00:37.830975685 5533 0x559d0acb2760 WARN h264parse gsth264parse.c:1349:gst_h264_parse_handleframe: broken/invalid nal Type: 1 Slice, Size: 165 will be dropped

Any ideas? Thank you very much in advance.

fduncanh commented 2 years ago

@shuco85

Yes! Finally a hint about the problem. The decryption and Airplay protocol is fine, but the video format being used by the mac seems to have changed.

I'll get back to you on this.

fduncanh commented 2 years ago

@shuco85 which version of gstreamer are you using?

shuco85 commented 2 years ago

GStreamer Core Library version 1.16.2

fduncanh commented 2 years ago

@shuco85 @amusingimpala75 @Booth1983

So now we know something about what is going on. It will be difficult until an appropriate macOS device is available here, so user input will be key. The essential problem may be a new choice of video format by the mac. This ought to be solvable with GStreamer. Luckily there appears to be no decryption problem, which would be much more difficult.

The uxplay -d option for debug logs shows nothing useful by itself . Instead, the GStreamer debug system will be key to solving this. 

export GST_DEBUG=2 ; uxplay -d

give warnings and errors. GST_DEBUG=4 gives much more (maybe too much) but is also very useful. Please post logs as files .

Here is a summary.

  1. Everything works on an Intel mac running macOS 10.x (catalina), which cannot run Big Sur or Monterey 11.x.
  2. We need to know if this is only a M1 (apple silicon) mac problem, or if it also is a problem on Intel macs running Big Sur or Monterey.
  3. We need to know if this is only a video problem, or if audio is also affected. Please look at a youtube video on the Mac and report whether the sound is rendered by UxPlay, even though the video is not.
  4. It seems that UxPlay itself is working correctly to set up the connection and decrypt the data stream from the mac. When this is done the two streams (audio) and video) are handed on to GStreamer pipelines to be rendered. Gstreamer is not understanding the format of the video stream it receives (is it still h264 format, or did Apple change format on M1 macs?). We will need to capture some of the video stream in order to examine the format. Some method for doing this will get added to Uxplay soon.
  5. It's possible that more recent versions of gstreamer (gstreamer1.0-plugins-bad) might understand the format (the error seems to be in parsing the h264 "NAL" by GStreamer. So it would be good to know whether the problem occurs also in GStreamer 1.20.0. The version of GStreamer varies with the OS and release. The problem is now known to occur for GStreamer 1.18.5 (Ubuntu 21.10) and earlier.

Thanks for any help in answering some of these questions. Please report the GStreamer version in any reports, and the operating system (e.g. Ubuntu 20.04LTS, etc)

maybe also explicity set the video sink with e.g,. -vs xvimagesink, -vs ximagesink, -vs glimagesink, -vs gtksink, etc. I don't think this will affect things, but explore whether the choice of videosink helps.

fduncanh commented 2 years ago

@shuco85

in GStreamer 1.16.2, it seems that the error message you see is that the "NAL" ("PPS" and SPS") which are h264 format information are not getting parsed. I am aware that UxPlay does not do something called "setting the caps" for gstreamer, but gets away with it (as programmed by the original author) , and it "just works" because GStreamer parses the PPS and SPS to find the "caps". With this new video stream it fails.

   if (!gst_h264_parse_process_nal (h264parse, &nalu)) {
      GST_WARNING_OBJECT (h264parse,
          "broken/invalid nal Type: %d %s, Size: %u will be dropped",
          nalu.type, _nal_name (nalu.type), nalu.size);
      *skipsize = nalu.size;
      h264parse->aud_needed = TRUE;
      goto skip;
    }
shuco85 commented 2 years ago

I could test UxPlay with a friend of mine's Mac BookPro 2015 (intel i7) with Monterey. It works like a charm.

With a Macbook Pro M1, the video doesn't work, but the audio does. I tried all the video sinks without success.

GST_DEBUG=4 gives too much irrelevant information before the connection, but when connecting, the logs are the same.

shuco85 commented 2 years ago

@shuco85

in GStreamer 1.16.2, it seems that the error message you see is that the "NAL" ("PPS" and SPS") which are h264 format information are not getting parsed. I am aware that UxPlay does not do something called "setting the caps" for gstreamer, but gets away with it (as programmed by the original author) , and it "just works" because GStreamer parses the PPS and SPS to find the "caps". With this new video stream it fails.

   if (!gst_h264_parse_process_nal (h264parse, &nalu)) {
      GST_WARNING_OBJECT (h264parse,
          "broken/invalid nal Type: %d %s, Size: %u will be dropped",
          nalu.type, _nal_name (nalu.type), nalu.size);
      *skipsize = nalu.size;
      h264parse->aud_needed = TRUE;
      goto skip;
    }

Excelent. So I just need to upgrade GStreamer, but I don't know how. I tried but there is no instructions on the web for Ubuntu, it just says "install GStreamer using your package manager". I also surfed the web but I couldn't find anything. Ubuntu only offers the 1.16 version with apt.

Any suggestion? Best regards

fduncanh commented 2 years ago

Unfortunately @amusingimpala75 reports "By the way, the server is being run on a mid-2011 iMac running Debian 11." which is 1.18.x , so it doesnt seem that upgrading GStreamer will solve it. Ubuntu 21.10 offers 1.18.x You probably have Ubuntu 20.04LTS which has 1.16.2. Its not easy to upgrade GStreamer unless you build from source, and until there is evidence that this fixes the issue, I would not recommend this. I will need to add something to UxPlay to save a copy of the "NAL", that you could then post to make it available. (I dont know too much about NAL's though...).

Booth1983 commented 2 years ago

A update to Gstreamer 1.18 doesn't seem to help. I just tried that...

fduncanh commented 2 years ago

@Booth1983

The problem seems to be the video format, but it will be solved once enough information is gathered.

let it stream for about a minute.

post the debug log so I can see what is going on with GStreamer.

Since I don't have a new mac, I am relying on reports from users like you.

fduncanh commented 2 years ago

The uxplay -d output now prints the h264 SPS (sequence parameter set) and PPS (picture parameter set) data. This will be useful to check what video format the Macbook M1 is sending.

Booth1983 commented 2 years ago

Here is the output for h264 SPS and PPS:

raop_rtp_mirror h264 Sequence Parameter Set:
27 64 00 2a ac 56 50 1e 00 89 f9 66 e0 20 20 c0 
da 08 84 65 80 

raop_rtp_mirror pps size = 4
raop_rtp_mirror h264 Picture Parameter Set:
28 ee 37 27

Further testing will have to wait until I reinstall my test system. Upgrade to Ubuntu 21.10 for gstreamer 1.18 has produced some problems with my setup. ;)

shuco85 commented 2 years ago

@fduncanh here you are my log if you need it debug_macos.log . same SPS and PPS as @Booth1983 !


raop_rtp_mirror sps size = 21
raop_rtp_mirror h264 Sequence Parameter Set:
27 64 00 2a ac             56 50 1e 00 89 f9 66 e0 20 20 c0   da 08 84 65 80 
  =  (base64)   J2QAKqxWUB4Aiflm4CAgwNoIhGWA
raop_rtp_mirror pps size = 4
raop_rtp_mirror h264 Picture Parameter Set:
28 ee 37 27                                              
 = (base64)   KO43Jw==
fduncanh commented 2 years ago

These are the values from a (unibody) mac running Catalina which works. some difference might be because it's non-retina.

raop_rtp_mirror sps size = 28
raop_rtp_mirror h264 Sequence Parameter Set:
27 64 00 29 ac              13 16 50 14 01 6e c0 1b 80 80 83 76 06 00 0b b8 00 02 ee 0b de f8 28   
 = (base 64) J2QAKawTFlAUAW7AG4CAg3YGAAu4AALuC974KA==

raop_rtp_mirror pps size = 4
raop_rtp_mirror h264 Picture Parameter Set:
28 ee 1f 2c                                      
 = (base64) KO4fLA==
fduncanh commented 2 years ago

From the SPS NALU ("sequence parameter set" "Network Abstraction Layer unit") old macs : 27 64 00 29 ac ....... = h264 "profile@level" = High@4.1 new macs: 27 64 00 2e ac ..... = h264 "profile@level" = High@4.2

where 27 means SPS, 64 = (decimal 100) means profile=High and 29 (= decimal 41) means level = 4.1 and 2e (= decimal 42) means level 4.2

Since the 4.1 level was from video streamed from an older 2012 unibody mac it could be that this difference is just the difference between Retina and non-Retina screen resolution, and the problem is buried in the other differences in parts of the PPS and SPS, which are a bit harder to decode ..... It would be very useful to get data from newer macs to see which work and which dont to pin this down. It could just be a bug in the GStreamer h264 parser that needs to interpret the PPA and SPS... Need to track down where the error in GStreamer occurs.

fduncanh commented 2 years ago

Here is the free pdf download of the standard ITU-T Rec H264 (06/2019) Advanced video coding for generic audio visual services. The table for interpreting SPS and PPS are in section 7.3.2.1 and 7.3.2.2 pages 44-45 (SPS) and pages 47-48 (PPS). A very useful commentary is provided in Yumi Chan's blog pages here and here. Also useful are this and this.

using h264_analyze from h264bitstream gives:

From the new M1 Macbook client (UxPlay broken)

!! Found NAL at offset 4 (0x0004), size 21 (0x0015) 
0.8: forbidden_zero_bit: 0 
0.7: nal->nal_ref_idc: 1 
0.5: nal->nal_unit_type: 7 
1.8: sps->profile_idc: 100          <-----------------profile = High (same as  Intel mac)
2.8: sps->constraint_set0_flag: 0 
2.7: sps->constraint_set1_flag: 0 
2.6: sps->constraint_set2_flag: 0 
2.5: sps->constraint_set3_flag: 0 
2.4: sps->constraint_set4_flag: 0 
2.3: sps->constraint_set5_flag: 0 
2.2: reserved_zero_2bits: 0 
3.8: sps->level_idc: 42              <------------level = 4.2  (Intel mac was 4.1)
4.8: sps->seq_parameter_set_id: 0 
4.7: sps->chroma_format_idc: 1 
4.4: sps->bit_depth_luma_minus8: 0 
4.3: sps->bit_depth_chroma_minus8: 0 
4.2: sps->qpprime_y_zero_transform_bypass_flag: 0 
4.1: sps->seq_scaling_matrix_present_flag: 0 
5.8: sps->log2_max_frame_num_minus4: 1    <---------- Intel mac was 8
5.5: sps->pic_order_cnt_type: 0 
5.4: sps->log2_max_pic_order_cnt_lsb_minus4: 2      <---------Intel mac was 10
5.1: sps->num_ref_frames: 4 
6.4: sps->gaps_in_frame_num_value_allowed_flag: 0 
6.3: sps->pic_width_in_mbs_minus1: 119            <--------------Intel mac was  was  107
8.6: sps->pic_height_in_map_units_minus1: 67  
9.1: sps->frame_mbs_only_flag: 1 
10.8: sps->direct_8x8_inference_flag: 1 
10.7: sps->frame_cropping_flag: 1                    
10.6: sps->frame_crop_left_offset: 0                  
10.5: sps->frame_crop_right_offset: 0               
10.4: sps->frame_crop_top_offset: 0                  
10.3: sps->frame_crop_bottom_offset: 4             
11.6: sps->vui_parameters_present_flag: 1 
11.5: sps->vui.aspect_ratio_info_present_flag: 0    <--- Intel Mac was 1, with additional aspect_ratio_info.
11.4: sps->vui.overscan_info_present_flag: 0 
11.3: sps->vui.video_signal_type_present_flag: 1 
11.2: sps->vui.video_format: 5 
12.7: sps->vui.video_full_range_flag: 1 
12.6: sps->vui.colour_description_present_flag: 1 
12.5: sps->vui.colour_primaries: 1 
13.5: sps->vui.transfer_characteristics: 1 
14.5: sps->vui.matrix_coefficients: 6 
15.5: sps->vui.chroma_loc_info_present_flag: 0   <------Intel mac was 1, with chroma_loc_info
15.4: sps->vui.timing_info_present_flag: 0 
15.3: sps->vui.nal_hrd_parameters_present_flag: 0    <----- Intel mac was 1, with nal_hrd_parameters given
15.2: sps->vui.vcl_hrd_parameters_present_flag: 0 
15.1: sps->vui.pic_struct_present_flag: 0            <----------Intel mac was 1
16.8: sps->vui.bitstream_restriction_flag: 1     <-------Intel mac was 0
16.7: sps->vui.motion_vectors_over_pic_boundaries_flag: 1    <------absent in Intel mac 
16.6: sps->vui.max_bytes_per_pic_denom: 2            <------absent in Intel mac                
16.3: sps->vui.max_bits_per_mb_denom: 1                <------absent in Intel mac                
17.8: sps->vui.log2_max_mv_length_horizontal: 16     <------absent in Intel mac                
18.7: sps->vui.log2_max_mv_length_vertical: 16         <------absent in Intel mac                
19.6: sps->vui.num_reorder_frames: 0                       <------absent in Intel mac              
19.5: sps->vui.max_dec_frame_buffering: 4                <------absent in Intel mac                 
20.8: rbsp_stop_one_bit: 1 
20.7: rbsp_alignment_zero_bit: 0 
20.6: rbsp_alignment_zero_bit: 0 
20.5: rbsp_alignment_zero_bit: 0 
20.4: rbsp_alignment_zero_bit: 0 
20.3: rbsp_alignment_zero_bit: 0 
20.2: rbsp_alignment_zero_bit: 0 
20.1: rbsp_alignment_zero_bit: 0 

!! Found NAL at offset 4 (0x0004), size 4 (0x0004) 
0.8: forbidden_zero_bit: 0 
0.7: nal->nal_ref_idc: 1 
0.5: nal->nal_unit_type: 8 
1.8: pps->pic_parameter_set_id: 0 
1.7: pps->seq_parameter_set_id: 0 
1.6: pps->entropy_coding_mode_flag: 1 
1.5: pps->pic_order_present_flag: 0 
1.4: pps->num_slice_groups_minus1: 0 
1.3: pps->num_ref_idx_l0_active_minus1: 0 
1.2: pps->num_ref_idx_l1_active_minus1: 0 
1.1: pps->weighted_pred_flag: 0 
2.8: pps->weighted_bipred_idc: 0 
2.6: pps->pic_init_qp_minus26: 0                <-----------------Intel mac was -1
2.5: pps->pic_init_qs_minus26: 0 
2.4: pps->chroma_qp_index_offset: -1             <-------------Intel mac was 0
2.1: pps->deblocking_filter_control_present_flag: 1 
3.8: pps->constrained_intra_pred_flag: 0 
3.7: pps->redundant_pic_cnt_present_flag: 0 
3.6: pps->transform_8x8_mode_flag: 1 
3.5: pps->pic_scaling_matrix_present_flag: 0 
3.4: pps->second_chroma_qp_index_offset: -1     <--------Intel mac  was 0
3.1: rbsp_stop_one_bit: 1

From the Intel Macbook client (UxPlay working) running Monterey .

!! Found NAL at offset 4 (0x0004), size 30 (0x001E) 
0.8: forbidden_zero_bit: 0 
0.7: nal->nal_ref_idc: 1 
0.5: nal->nal_unit_type: 7 
1.8: sps->profile_idc: 100 
2.8: sps->constraint_set0_flag: 0 
2.7: sps->constraint_set1_flag: 0 
2.6: sps->constraint_set2_flag: 0 
2.5: sps->constraint_set3_flag: 0 
2.4: sps->constraint_set4_flag: 0 
2.3: sps->constraint_set5_flag: 0 
2.2: reserved_zero_2bits: 0 
3.8: sps->level_idc: 41 <-------- level = 4.1   (M1 mac  was 4.2)
4.8: sps->seq_parameter_set_id: 0 
4.7: sps->chroma_format_idc: 1 
4.4: sps->bit_depth_luma_minus8: 0 
4.3: sps->bit_depth_chroma_minus8: 0 
4.2: sps->qpprime_y_zero_transform_bypass_flag: 0 
4.1: sps->seq_scaling_matrix_present_flag: 0 
5.8: sps->log2_max_frame_num_minus4: 8  <---------------M1 mac was 1
5.1: sps->pic_order_cnt_type: 0 
6.8: sps->log2_max_pic_order_cnt_lsb_minus4: 10 <----------M1 mac was 2
6.1: sps->num_ref_frames: 4 
7.4: sps->gaps_in_frame_num_value_allowed_flag: 0 
7.3: sps->pic_width_in_mbs_minus1: 107 <----M1 mac was 119
9.6: sps->pic_height_in_map_units_minus1: 67 
10.1: sps->frame_mbs_only_flag: 1 
11.8: sps->direct_8x8_inference_flag: 1 
11.7: sps->frame_cropping_flag: 1 
11.6: sps->frame_crop_left_offset: 0 
11.5: sps->frame_crop_right_offset: 0 
11.4: sps->frame_crop_top_offset: 0 
11.3: sps->frame_crop_bottom_offset: 4 
12.6: sps->vui_parameters_present_flag: 1 
12.5: sps->vui.aspect_ratio_info_present_flag: 1  <---M1 mac was 0
12.4: sps->vui.aspect_ratio_idc: 0  <---- absent in M1 mac
13.4: sps->vui.overscan_info_present_flag: 0 
13.3: sps->vui.video_signal_type_present_flag: 1 
13.2: sps->vui.video_format: 5 
14.7: sps->vui.video_full_range_flag: 0 
14.6: sps->vui.colour_description_present_flag: 1 
14.5: sps->vui.colour_primaries: 1 
15.5: sps->vui.transfer_characteristics: 1 
16.5: sps->vui.matrix_coefficients: 1 
17.5: sps->vui.chroma_loc_info_present_flag: 1    <-----M1 mac was 0
17.4: sps->vui.chroma_sample_loc_type_top_field: 1 <----absent in M1 mac
17.1: sps->vui.chroma_sample_loc_type_bottom_field: 1 <---- absent in M1 mac
18.6: sps->vui.timing_info_present_flag: 0 
18.5: sps->vui.nal_hrd_parameters_present_flag: 1  <-- M1 mac was 0
18.4: sps->hrd.cpb_cnt_minus1: 0   <--- absent in M1 mac
18.3: sps->hrd.bit_rate_scale: 0   <----absent in M1 mac
19.7: sps->hrd.cpb_size_scale: 3   <----absent in M1 mac
19.3: sps->hrd.bit_rate_value_minus1[ SchedSelIdx ]: 11999 <---absent in M1 mac
23.8: sps->hrd.cpb_size_value_minus1[ SchedSelIdx ]: 5999 <--- absent in M1 mac
26.7: sps->hrd.cbr_flag[ SchedSelIdx ]: 0  <--- absent in M1 mac
26.6: sps->hrd.initial_cpb_removal_delay_length_minus1: 23  <---absent in M1 mac
26.1: sps->hrd.cpb_removal_delay_length_minus1: 23 <-----absent in M1 mac
27.4: sps->hrd.dpb_output_delay_length_minus1: 23 <---absent in M1 mac
28.7: sps->hrd.time_offset_length: 24 <---absent in M1 mac
28.2: sps->vui.vcl_hrd_parameters_present_flag: 0 
28.1: sps->vui.low_delay_hrd_flag: 0 <---absent in M1 mac
29.8: sps->vui.pic_struct_present_flag: 1 <--- 0 in M1 mac
29.7: sps->vui.bitstream_restriction_flag: 0 
29.6: rbsp_stop_one_bit: 1 
29.5: rbsp_alignment_zero_bit: 0 
29.4: rbsp_alignment_zero_bit: 0 
29.3: rbsp_alignment_zero_bit: 0 
29.2: rbsp_alignment_zero_bit: 0 
29.1: rbsp_alignment_zero_bit: 0
fduncanh commented 2 years ago

@shuco85

I could test UxPlay with a friend of mine's Mac BookPro 2015 (intel i7) with Monterey. It works like a charm. With a Macbook Pro M1, the video doesn't work, but the audio does. I tried all the video sinks without success.

Thanks for this info! It could be useful to look at the SPS and PPS from that intel Mac running Monterey, if possible (to see what changes from Catalina are dues to Monterey in general, and which are M1 specific). (Now I finally learned how to extract their info..)

_GSTDEBUG=4 gives too much irrelevant information before the connection, but when connecting, the logs are the same. The logs will be similar, but not identical. They now contain the SPS and PPS info.

Booth1983 commented 2 years ago

Has anyone tried running uxplay with predefined resolution? I couldn't reinstall my system this weekend but I'll get it ready at the beginning of this week. Then I can start testing again with the M1.

I'm really no expert, but it seems to me, that die M1 has problems or works differently with resolution and frame rate. On the other hand I'll keep to you pros.

I do have to say: Thanks for all the time invested!

fduncanh commented 2 years ago

Yes It might be worth testing with -s(width)x(height).

If I get the PPS and SPS from @shuco's friend's intel MAC running Monterey It will help to narrow down what the M1 does differently.

shuco85 commented 2 years ago

Yes It might be worth testing with -s(width)x(height).

If I get the PPS and SPS from @shuco's friend's intel MAC running Monterey It will help to narrow down what the M1 does differently.

Here you have:

raop_rtp_mirror h264 Sequence Parameter Set:
27 64 00 29 ac 13 16 50 1b 00 89 f9 70 06 a0 20 
20 34 98 18 00 2e e0 00 0b b8 2f 7b e0 a0 

raop_rtp_mirror pps size = 4
raop_rtp_mirror h264 Picture Parameter Set:
28 ee 1f 2c

I tried -s(width)x(height) but it doesn't work

Thank you guys for your interest

fduncanh commented 2 years ago

@shuco85

Thanks! The only difference between the old unibody intel mac running Catalina and the newer retina Intel mac running Monterey turned out to be screen width and height in pixels and some screen cropping features: otherwise essentially the same.

The most obvious change in the M1 is that the h264 High profile "level" went up from 4.1 to 4.2

fduncanh commented 2 years ago

News: An M1 mac will soon be available, so we will hopefully be able to debug this GStreamer issue. It may require a patch to fix GStreamer. EDIT: now have a M1 mac to use for debugging this issue.

raop_rtp_mirror accepting client
begin video stream wxh = 1920x1080; source 1920x1080
raop_rtp_mirror width_source = 1920.000000 height_source = 1080.000000 width = 1920.000000 height = 1080.000000
raop_rtp_mirror sps size = 21
raop_rtp_mirror h264 Sequence Parameter Set:
27 64 00 2a ac 56 50 1e 00 89 f9 66 e0 20 20 c0 
da 08 84 65 80 

raop_rtp_mirror pps size = 4
raop_rtp_mirror h264 Picture Parameter Set:
28 ee 37 27 

raop_rtp_mirror video ntp = 1648770303549904, now = 1648770303526407, latency = -23497
Begin streaming to GStreamer video pipeline
0:00:13.710848480  7064      0x1779630 WARN               h264parse gsth264parse.c:1351:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 6 SEI, Size: 30 will be dropped
0:00:13.710899533  7064      0x1779630 WARN               h264parse gsth264parse.c:1351:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 5 Slice IDR, Size: 68164 will be dropped
raop_rtp_mirror video ntp = 1648770303583237, now = 1648770303533835, latency = -49402
0:00:13.718009799  7064      0x1779630 WARN               h264parse gsth264parse.c:1351:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 2062 will be dropped
raop_rtp_mirror video ntp = 1648770303616570, now = 1648770303540722, latency = -75848
0:00:13.724906985  7064      0x1779630 WARN               h264parse gsth264parse.c:1351:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 20447 will be dropped
fduncanh commented 2 years ago

Fixed. It was a simple UxPlay error, not a GStreamer one. The original author (dsafa22) of the Airplay Mirror code assumed that the next NAL unit (video data) after the SPS and PPS NAL's (the two format descriptors) was true 'VCL' video data. The M1 macs went from h264 level High@4.1 to High@4.2, which added a third NAL of descriptive data before the first video NAL arrives. This broke the assumption made in the old code. (RPiPlay does not have this error, the error came from a later version of dsafa22's code.) The new code is now robust against this and future similar changes by Apple.

@shuco85 @amusingimpala75 @Booth1983 please test and report if the fix is working for you.

Booth1983 commented 2 years ago

Finally I was able to reinstall my system. Thanks for your time and effort. Image ist opening now. Colors aren't correct, but that's probably because of my sink settings on my test system. I'll try to fix that later. => Other sink fixed that problem.

I think I have to write in English more often to get my grammar and vocabulary in shape again.

shuco85 commented 2 years ago

Now everything works like a charm. Thank you very much to all of you for the effort.

Best Regards.

Booth1983 commented 2 years ago

After I tested on several PCs here, I can say everything works Great! That includes Mac M1 and Intel, IPads and IPhones.

Thank you very much :-)

fduncanh commented 2 years ago

you're welcome!
thanks for providing feedback, both when things work and when they don't