search

Kurento / bugtracker

[ARCHIVED] Contents migrated to monorepo: https://github.com/Kurento/kurento
46 stars 10 forks source link

Record video get stuck after orientation changed #484

Open vanhungoz opened 4 years ago

vanhungoz commented 4 years ago
    • I have read the SUPPORT document
    • I have checked the Troubleshooting Guide
    • I have tested with the latest version of Kurento

i am using kurento version 6.13 to record a streaming video. whenever i rotate the screen, video always get stuck. it always displays the latest image of the previous orientation. below is a video record sample.

https://corona-s3fs.s3-ap-northeast-1.amazonaws.com/96cb8a3c-9faa-48b5-8bf0-3a75025901bc.MP4

vanhungoz commented 4 years ago

the client is using react-native-webrtc

j1elo commented 4 years ago

This maybe is: a) A limitation of the RecorderEndpoint: ~once the recording has started, changes in video resolution size are not accepted~. UPDATE: Some testing proved that this is not true, and changes in resolution are correctly handled by the RecorderEndpoint. b) A limitation of the container format (MP4, MKV, WEBM) c) An issue introduced by Safari/iOS.

Related issues:

Related topics:

This is a way the issue has manifested before, with video coming from an iOS Safari, when the user rotated the device:

First format detected for GStreamer capabilities is like this:

video/x-h264, stream-format=(string)avc, [...], width=(int)480, height=(int)640, framerate=(fraction)0/1

then it suddenly changes to this...

video/x-h264, stream-format=(string)avc, [...], width=(int)640, height=(int)480, framerate=(fraction)0/1

...and then it is when the error message appears: pad video_0 refused renegotiation.

Note the chage of width and height. Other samples of this issue included a change in total video size upon rotating the device, so it is not only a matter of swapping width with height.

j1elo commented 4 years ago

I had forgotten, but from this comment maybe the problem is not RecorderEndpoint, but the MP4 container format that is not compatible with dynamic changes in video resolution.

Users affected by this issue should try recording with MP4, WEBM, and MKV formats, and let us know if the issue is solved with any of those formats, so we can update the RecorderEndpoint documentation to reflect that fact.

j1elo commented 4 years ago

I have been able to test with an iPhone 6 with iOS 12.4. The result is that rotation causes a stuck video in all 3 recording formats (WEBM, MKV, MP4).

I also tested with Chrome and Firefox on Android. This is a summary:

What is even more important: ffplay says that there is no video data whatsoever during the stuck periods. During the time the phone was rotated and the recorded video hangs, there is actually no video data in the recorded file. So that's why it looks like it's stuck.

Capture of the ffplay command. Note how the timestamps come to a halt at 00:05:23, then it resumes at 00:11:00. That's from the time the iPhone was rotated 90 degrees.

asciicast

I've been thinking about that and this might be caused by video frames not flowing between the WebRtcEndpoint and the RecorderEndpoint. But why? Is the iPhone actually sending video while the phone is rotated? (because if not, that's clearly an iOS issue).

j1elo commented 4 years ago

Adding to

Is the iPhone actually sending video while the phone is rotated?

I've set up a simple test with the OneToOne tutorial, between Safari/iOS and Chrome desktop.

The result is that the video from Safari also got stuck in the Chrome page. Chrome is not receiving input video when the iPhone is rotated!! This takes the RecorderEndpoint out of the issue.

Now the question is: why is Chrome not receiving video? Is it because Kurento has a bug and it stops sending such video? Or is it because iOS is the one with a bug and it stops sending?

micaelgallego commented 4 years ago

The conclusion is that webrtc video from iOS Safari can not be managed by Kurento when orientation is changed or iOS itself can not manage it?

A test with a other WebRtc service like whereby should clear the doubts.

j1elo commented 4 years ago

I guess the conclusion is that, when rotating, Safari does something with the way it encodes videothat is very different to what Chrome & Firefox do. Kurento was optimized to work with these two, and it doesn't like whatever Safari does.

Maybe it is how the H.264 stream is reconfigured, or how the keyframes are emitted. H.264 is not only video, but also metadata inbetween, so there is a lot that could be going on, and to which the decoder in Kurento can react better or worse. Now that I think of it, maybe upgrading the decoder (GStreamer) would have an impact on this issue.

micaelgallego commented 4 years ago

I understand... please add a note somewhere in RecorderEnpoint until we can spend time to investigate the problem further and try to fix it.

j1elo commented 4 years ago

I found another source that makes me think this is, after all, a bug in Safari for iOS ≤ 13.0:

@vanhungoz what version of iOS have you tested? Can you compare between 12.x and 13.x?

vanhungoz commented 4 years ago

@j1elo sorry for my late. i am using ios 12.4.6

vanhungoz commented 4 years ago

client is app that is developed by react-native

vanhungoz commented 4 years ago

now, i need to update xcode to test ios 13.5. will be notify you when i done

vanhungoz commented 4 years ago

@j1elo this is some of my test: MP4 + ios 12.4.6 and 13.3.1: the result is a video that hangs whenever the phone is rotated. WEBM + ios 12.4.6 and 13.3.1 and 13.5.1: it work

RaistGH commented 4 years ago

Hi,

Just as a note, maybe it's not just a Safari issue. Remember this issue where Windows+Chrome is changing resolution after few seconds. The Safari version we tested was 12.1. Maybe new Safari versions are doing the same. Whenever we get some budget I'll try to do new checks with WEBM and MKV containers to see if they solve also the problem for Windows+Chrome since now iOS can support containers other than MP4.

Regards

AlexandrDiakov commented 2 years ago

Are there any news or updates regarding this issue or whats causing it? We are experiencing similar problem with video playback of incoming remote stream freezing after rotating iOS device (tested with 15.2.1) but it is inconsistent, sometimes it happens and sometimes not. Also, video freezes not upon first rotation but when user flips device back to its original orientation which makes me think it might be completely another problem. Any thoughts would be much appreciated.

j1elo commented 2 years ago

Are there any news or updates regarding this issue or whats causing it? We are experiencing similar problem with video playback of incoming remote stream freezing after rotating iOS device (tested with 15.2.1) but it is inconsistent, sometimes it happens and sometimes not. Also, video freezes not upon first rotation but when user flips device back to its original orientation which makes me think it might be completely another problem. Any thoughts would be much appreciated.

Please open a separate bug report if you have observed these issues and are not related with the recording. This bug report is for the RecorderEndpoint.

neilyoung commented 1 year ago

The problem is in qtmux or mp4mux, which refuses renegotiation of caps if changed downstream (e.g. by resolution change).

It is even present, if not using Recorderendpoint, but RtpEndpoint and use a GST pipeline for the file save.

0:00:11.043992344 22062 0x7ff1100eb280 WARN                   qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<qtmux0> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0142c020ffe1000f6742c0208c8d407808bf700f08846a01000468ce3c80, level=(string)3.2, profile=(string)constrained-baseline, width=(int)960, height=(int)540, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true