Only the first frame is displayed (Android)

geovlet commented 1 year ago

Describe the bug If you run the example project in Android, only the first frame is shown. (played with RTSP protocol in VLC player)

To Reproduce Steps to reproduce the behavior:

Build the example project on Android
Send RTMP stream to Wowza media server
Testing playback with RTSP protocol in VLC

Smartphone (please complete the following information):

Device: Samsung Galaxy S10, S10+, S10 Note
OS: Android 12
Version: api.video-flutter-live-stream 1.0.7

ThibaultBee commented 1 year ago

Hi,

Could you contact Wowza to get info/debug log? Have you already contacted the Wowza support? There is not enough information to debug something.

FYI, we only tested this library with api.video and ffplay as servers. We haven't tested on other platform, so you might face issues if you try a live stream to another platform.

Vis5 commented 1 year ago

Same issue. I'm getting these two errors from the logs: "Timecode out of order" and "Timecodes jumped back in time".

ThibaultBee commented 1 year ago

Ok. Could you explicit the steps to reproduce the behavior? What server are you trying to send frame?

Vis5 commented 1 year ago

Pretty much the same steps as the OPs.

Build the example project on Android
Send RTMP stream to Wowza media server
Testing playback with HLS protocol in VLC (only difference here is I'm using HLS for playback instead of RTSP but that's irrelevant as the issue happens during steps 1 and 2)

Device: Samsung Galaxy S8 OS: Android 10 Version: api.video-flutter-live-stream 1.0.7

The exact logs are as below:

2022-11-27 00:57:46 EST comment server INFO 200 - LiveStreamPacketizerCupertino.handlePacket[live/_definst_/kinno19_1669528636_160p][mp4a.40.2]: AAC Audio info: {AACFrame: codec:AAC, channels:2, frequency:44100, samplesPerFrame:1024, objectType:LC} - - - 9154910.697 - - - - - - - - - - - - - - - - - - - - - - - - - 2022-11-27 00:57:46 EST comment server INFO 200 - LiveStreamPacketizerCupertino.handlePacket[live/_definst_/kinno19_1669528636_160p][avc1.66.21]: H.264 Video info: {H264CodecConfigInfo: codec:H264, profile:Baseline, level:2.1, frameSize:90x160, displaySize:90x160, frameRate:30.0, PAR:1:1, crop: l:0 r:3 t:0 b:0} - - - 9154910.697 - - - - - - - - - - - - - - - - - - - - - - - - - 2022-11-27 00:57:46 EST comment server WARN 200 - LiveStreamPacketizerPacketHandler.handlePacket[live/_definst_/kinno19_1669528636]: Timecode out of order [video]: 4294967232:4294967239 - - - 9154910.781 - - - - - - - - - - - - - - - - - - - - - - - - - 2022-11-27 00:57:46 EST comment server WARN 200 - LiveStreamPacketizerPacketHandler.handlePacket[live/_definst_/kinno19_1669528636]: Timecode out of order [audio]: 4294967316:4294967336 - - - 9154910.794 - - - - - - - - - - - - - - - - - - - - - - - - - and 2022-11-27 00:57:46 EST comment server WARN 200 - LiveStreamPacketizerPacketHandler.resetStream[live/_definst_/kinno19_1669528636][11:0]: Timecodes jumped back in time. - - - 9154910.875 - - - - - - - - - - - - - - - - - - - - - - - - - 2022-11-27 00:57:46 EST comment server WARN 200 - LiveStreamPacketizerPacketHandler.handlePacket[live/_definst_/kinno19_1669528636]: Timecode out of order [audio]: 4294967316:4294967396 - - - 9154910.875 - - - - - - - - - - - - - - - - - - - - - - - - - 2022-11-27 00:57:46 EST comment server WARN 200 - LiveStreamPacketizerPacketHandler.resetStream[live/_definst_/kinno19_1669528636][11:0]: Timecodes jumped back in time. - - - 9154910.876 - - - - - - - - - - - - - - - - - - - - - - - - -

I've tried some solutions provided by the wowza side but so far no luck :(

https://www.wowza.com/docs/how-to-fix-unaligned-video-and-audio-with-a-server-side-sort-buffer

https://www.wowza.com/docs/how-to-turn-on-an-rtp-jitter-buffer-and-packet-loss-logging-rtp-and-mpeg-ts

ThibaultBee commented 1 year ago

Ok, I think I know where it comes from.

In our Android side, internaly, our video and our audio streams are not synchronized. The audio encoder encodes frames a lot faster than the video encoder. The result of this behavior is that older video frames arrive later than the newer audio frame. A drawing would be better to explain this but what you have to understand is that audio and video frames are out of order.

Strict RTMP server expects to have frames order by timestamp because it is a requirement of the RTMP. Unfortunately, the order of audio and video frames is really difficult to achieve without:

synchronising audio and video encoders: but if one of them is stuck the other would be stuck as well. It might not be great for performances
changing the timestamp (cheapest version) (so they are always in order - but the audio and video frames would not be synchronised - this is the solution of most Android RTMP library).
complex processes where the main idea is to wait for video frames but it has lot of drawbacks and you don't want to run these algorithms on phone that already have a lot to handle with the live stream.

We have tested our library with ffmpeg, api.video and also youtube and it works perfectly. I am sorry to say this but we don't have an easy solution for you to run on wowza. I am not familiar with Wowza but maybe you can find a way to fix that on the server side or you might contact them.

Best regards, Thibault

Vis5 commented 1 year ago

I see. Thank you for the write-up! One small question: How do ffmpeg, api.video, youtube etc. stream both audio and video synchronized without knowing the order? Are they just expecting there would be no significant delay between the audio and the video? or are they waiting on the audio frames until the video frames arrive?

ThibaultBee commented 1 year ago

I am not familiar with the server side but the timestamp of each frame is passed in the RTMP/FLV header. I am not sure exactly, I guess there are just buffering frames for X ms.

Vis5 commented 1 year ago

Yeah, I'm guessing they do some kind of magic like that. Regardless, thank you for the help! Seems like I'm out of luck as this issue won't be supported.

ThibaultBee commented 1 year ago

Could you test with v1.1.0?

Vis5 commented 1 year ago

Woohoo, glad to see it being worked on. I can actually watch the video feed now. However it's very laggy. As in, it's like I'm watching it in 1 fps (I suspect that's the keyframes inserted every second). I couldn't test if the audio and video were in sync but I can tell audio was playing.

The logs are basically the same as before:

CupertinoPacketHandler.resetStream[live/_definst_/1673504108_360p][11:2]: Timecodes jumped back in time.

LiveStreamPacketizerPacketHandler.handlePacket[live/_definst_/1673504108_360p]: Timecode out of order [video]: 4294967329:4294967396

CupertinoPacketHandler.handlePacket[live/_definst_/1673504108]: Timecode out of order [audio]: curr:4294967290 last:4294967403 diff:-113 maxTimecodeGap:-1

ThibaultBee commented 1 year ago

Hi, I added lot of logs in a specific branch of the Android live stream library. Could you install the demos-camera apk in https://github.com/ThibaultBee/StreamPack/suites/10351101299/artifacts/510056555? Stream your video and send me the logs?

Vis5 commented 1 year ago

Hello, I somehow didn't receive notification for this (most likely I received it and glossed over it). Anyway, I've downloaded the apks you sent and tried live streaming. But where do I get the logs from?

ThibaultBee commented 1 year ago

From adb logcat.

If you are using Android Studio, there is a Logcat panel in lowset part.
with the cli, adb logcat

Vis5 commented 1 year ago

ooh I see, of course!

Here are the logs: https://justpaste.it/cwux6

ThibaultBee commented 1 year ago

Hi,

From the logs, I notice that newer AAC (audio) frames are received by the RTMP publisher later than older H.264 (video) frames. The only reasons why it oculd be happening are:

H.264 encoder is really fast and optimized in this device
Somehow a lock slow AAC stream

As audio and video frames are generated in 2 differents threads, they comes out of encoders without any orders. I implemented an algorithm that wait for the next video frame to send audio frames. Anyway, the algorithm I implemented was not expected that video frames come faster than audio frames.

I have no solution for now and not being able to reproduce it on my own does not facilitate the debug.

Could you reproduce with other devices?

Vis5 commented 1 year ago

Video is faster than the audio? That's really interesting indeed. I didn't expect that either. I'll check with other devices and let you know.

ThibaultBee commented 7 months ago

Hi, Sorry for the delay, I know it has been a while but I have been busy on 100 stuff.

With a new phone, I could reproduce a similar issue and I did made a correction. Could you check with https://github.com/ThibaultBee/StreamPack/suites/18441384366/artifacts/1069203816?

Vis5 commented 7 months ago

wow it's been a while. Thanks, I don't have access to a wowza server anymore but I'll try.

ThibaultBee commented 7 months ago

Sorry I still need to work on this subject. It is not yet done.

ThibaultBee commented 7 months ago

Seems better there with these artifacts: https://github.com/ThibaultBee/StreamPack/suites/18448322708/artifacts/1069695034

Unfortunately, I did not find any other solution than delay the packet by 100 ms in order to synchronise audio and video. I am not sure 100 ms is enough but it is already a lot.

ThibaultBee commented 5 months ago

I made changes on how to handle the frame with timestamp not in order in the main branch. Could you test it on wowza please?

ThibaultBee commented 2 months ago

Stalled issue. Please reopen if you face it again.

apivideo / api.video-flutter-live-stream

Only the first frame is displayed (Android) #18