search

RenderHeads / UnityPlugin-AVProVideo

AVPro Video is a multi-platform Unity plugin for advanced video playback
https://www.renderheads.com/products/avpro-video/
231 stars 28 forks source link

Small videos freezing #1944

Open Anton111111 opened 1 month ago

Anton111111 commented 1 month ago

I have grid with video cards and when user hover card i add a new instance of mediaplayer and DisplayUGUI and Load a small thumbnail video. But some times video freezing. This code works good with AVPro v2 without any freezings. And if i change exoplayer to mediaplayer (in AVPro v3) freez gone away. I created a small project to learn this issue and after many tests and different videos i see that when video is freezed buffer is full. Why it doesn't start i don't know. I also see this problem on very small videos up to 100kb. The faster the video loads, the more often the problem appears. In attached video you can see issue. You can see events logs and buffer state (i write MinTime and MaxTime from TimeRanges that returned from GetBufferedTimes() in lower line) on this video.

Your Setup (please complete the following information): Unity version: 2023.1.20f1 AVPro Video version (number and edition (trial/core/ultra/enterprise)): 3.0.6-ultra build 2407050949 Operating system version: Android Device model: Quest2/Quest3

Videos https://github.com/RenderHeads/UnityPlugin-AVProVideo/assets/987933/dccdd455-fea4-4ece-aed4-2e9561cfef6f

Ste-RH commented 1 month ago

We are going to need a full unfiltered logcat file at very least.

Anton111111 commented 1 month ago

@Ste-RH , There can be a maximum of two players at the same time. But more often than not there is only one. After leaving the card, I call CloseMedia() and destroy game object with the mediaplayer component. I don’t think it’s a matter of system resources because the size of played thumbnail video is 600-800 kilobytes. And i did see this issue with the same code in AVPro v2.

I changed to locally video and got worst result. It is freezed more frequntly. Here is log (with loading locally video AVProVideoSamples/4260_300p.mp4): odh_logs_2024-07-11 13.48.14.438.txt

Do you want check my test project ?

Ste-RH commented 1 month ago

Do you want check my test project ?

I think that would be very helpful, thanks. Do send it over to unitysupport@renderheads.com with the subject 'Issue #1944'

Ste-RH commented 1 month ago

And i did see this issue with the same code in AVPro v2.

Just to double check. You did see the same issue in v2?

Anton111111 commented 1 month ago

@Ste-RH , I'v sent an invite to a project in github. Please, ignore the name of the project because i've created it for another issue. Currently this project is related to Issue #1944 and Issue #1934

I don't see this issue in v2. This code works good on v2.

Ste-RH commented 1 month ago

I noticed in your scene you have a MediaPlayer that is loading the sample 'cones' video. When I disabled that MediaPlayer, videos went from playing back 19 times out of 20, to 1 in 20 !

With that failure rate, I tried using the BigBuckBunny sample video that we send out with AVPro Video. It is approximately 8 seconds in length, shorter than your video, and I did not see a video fail to open once. The primary difference between that video and yours being there is an audio track in the BBB video. I added a silent audio track to your video and it plays back flawlessly for me. I will send that test video over to you via email.

I think the MediaPlayer you have in the scene essentially 'keeps ExoPlayer around/initialised'. At least that is my hunch for why it got very bad when I disabled it.

I will have to take your word for it working in v2, but that in itself is odd. We did switch to using the newer media3 versionn of ExoPlayer when we moved to v3, so maybe there is some bug/difference we are now seeing with shorter videos with no audio track?

Anyway, if you can put an audio track into your videos to test...do let us know. Oh, yeah, also...if you are going to be pulling videos off a cloud server then you absolutely want to have the MOV atom at the start of your videos. The one in your sample project has the MOV atom at the end, which means the whole video has to be downloaded off the server before playback can start.

Anton111111 commented 1 month ago

I've made the following to test it. I've removed at all mediaplayer and sphere with cones. I've got the same results. Video stacks. And i've added your video with audio. And i can confirm that no issue with video with audio. Also i've tested it again with avpro v2 and i see that no problem with any video (with audio or without).

You can see two branches in my project:

Unfortunately, I cannot change the video format that will be used for video thumbs.

Anton111111 commented 1 month ago

In additional i made another test. I've got media3 repo (https://github.com/androidx/media/tree/1.1.0). How i understand you use tag 1.1.0 and a little bit rework demo-surface to init exoplayer on each change surface and changed video url to https://cdn-vr.deovr.com/preview/14x1/4260_300p.mp4.I installed builded apk to quest2 and i don't see issue with video start. Exoplayer reinited many times and video reloaded many times and it always start without problems.

I understand that this is not a pure experiment. But maybe this will help you.

Ste-RH commented 1 month ago

How i understand you use tag 1.1.0 and a little bit rework demo-surface to init exoplayer on each change surface

Not in the slightest. We are completely custom code through the entirety of the plugin.

This is a very strange issue and we are currently at a loss as to what is causing it. Like you, I checked and v2 seems to not suffer it.

I am able to reproduce it now outside your app, and have logging side by side of when the video plays and when it does not. There is one callback that does not arrive from ExoPlayer when the video does not play : onVideoSizeChanged(...) and also the ExoPlayer state seems to sit in 'buffering'. I have noticed that if I call a seek then playback is then fine.

The strangest thing about all this is that you (and me in my test project) are killing off the player completely. This internally kills off the ExoPlayer instance and then a new one is created when the new/next player is created. Next thing I am going to try is a pause between destroying the old and creating the new player. [Edit]: Nope, that did not help.

Anton111111 commented 1 month ago

Not in the slightest. We are completely custom code through the entirety of the plugin.

I meant you use jetpack Media3 version 1.1.0 (https://developer.android.com/jetpack/androidx/releases/media3#1.1.0)

Ste-RH commented 1 month ago

Not in the slightest. We are completely custom code through the entirety of the plugin.

I meant you use jetpack Media3 version 1.1.0 (https://developer.android.com/jetpack/androidx/releases/media3#1.1.0)

As the core player API, yes. You can also switch to use the MediaPlayer component to use the Android MediaPlayer API. The issue does not happen for me using this.

Anton111111 commented 1 month ago

As the core player API, yes. You can also switch to use the MediaPlayer component to use the Android MediaPlayer API. The issue does not happen for me using this.

I know about Android MediaPlayer API. I wrote about it in the issue description.

Ste-RH commented 1 month ago

I went digging into ExoPlayer (media3) issues and found this:

https://github.com/androidx/media/issues/1132

...that led me to this:

https://github.com/google/ExoPlayer/issues/10021#issuecomment-1134589796

...which is actually an option we support:

image

However, this appears to be a little bit broken in v3.x, though I am not sure it is 100% AVPro's fault! Enabling that spat a missing class error: java.lang.NoClassDefFoundError: Failed resolution of: Landroidx/collection/CircularIntArray;

To remove this error I had to, in Unity, enable these two build properties:

image

...and add the following lines to the templates:

image

image

Not ideal, and not really sure what we can do about it short of including androidx/collections in the AVPro Video asset package...which would likely throw issues for other developers out there using a different version of that library. Suggestions welcome!

Give it a try and let me know what you get @Anton111111.

Anton111111 commented 1 month ago

@Ste-RH , i've tried enable forceEnableMediaCodecAsynchronousQueueing. It doesn't fix issue at all. And i don't see class error: java.lang.NoClassDefFoundError: Failed resolution of: Landroidx/collection/CircularIntArray after enabling.

Also in additional. How i see i get ResolutionChanged event when video stucks.

Ste-RH commented 1 month ago

Hmm. If you are not getting the error, maybe it is not enabling as it should, or maybe you have the library pulled in by something else in your project?

I was testing on a Xiaomi Mi 10T though. Will try on a Quest 2. I am 90% sure this is the issue.

Another way you could address the problem in your project is to not continually create/destroy MediaPlayers, but instead have a pool of them (likely only needing 2) that you pick from and open the video/assign the player accordingly. Then close it on change and pull a different player for the next tile. This would be a hell of a lot better in terms of memory/GC as well.

Anton111111 commented 1 month ago

@Ste-RH , Initially, I had one mediaplayer that I reused and there was exactly the same problem.

Currently i added _mediaPlayer?.Control?.SeekToFrame(0); when i get event MetaDataReady. And looks like this trick fixes issue.

Ste-RH commented 1 month ago

A pool of players would be the way I went about it...and I don't see the problem at all here in our testbed (that hammers through opening/closing videos as well as destroying/creating if desired) with open/close.

Ste-RH commented 1 month ago

@Anton111111 I have just emailed over to you a screen recording of it on Quest 2. It is working exactly as I think it should. This is with the forceEnableMediaCodecAsynchronousQueueing flag enabled on the MediaPlayer in the 'Prefab' GameObject in your test scene. Without it, I get a lot of none video loads.

Anton111111 commented 1 month ago

@Anton111111 I have just emailed over to you a screen recording of it on Quest 2. It is working exactly as I think it should. This is with the forceEnableMediaCodecAsynchronousQueueing flag enabled on the MediaPlayer in the 'Prefab' GameObject in your test scene. Without it, I get a lot of none video loads.

Can you push your changes? I build again with forceEnableMediaCodecAsynchronousQueueing enabled but it still doesn't work.

Ste-RH commented 1 month ago

We have released v3.0.7 a few hours ago. Maybe something in the latest code-base helps along with the flag enabled? Update and give it a go.

Anton111111 commented 1 month ago

We have released v3.0.7 a few hours ago. Maybe something in the latest code-base helps along with the flag enabled? Update and give it a go.

I've checked with 3.0.7. I still see issue with forceEnableMediaCodecAsynchronousQueueing enabled. I've pushed after update changes to video-without-audio branch

Ste-RH commented 1 month ago

This is very strange. As you can see in the video I emailed over to you, I do not get the issue at all when I enable that flag. Are you enabling it in the editor on the MediaPlayer component, or in script?

Ste-RH commented 1 month ago

Aghh. My bad. Just noticed I was actually loading the video that I had put an audio track into. So yeah, that flag does not help :/

Anton111111 commented 2 days ago

@Ste-RH , Any news?

Ste-RH commented 9 hours ago

No, afraid nothing new to report yet. I hoped that a newer version of media3 would offer something, but having tried 1.4.1 it does not change anything - only brings a dependency issue such that we do not want to update to it's use just yet in AVPro Video.

What I did notice is that when a video fails to load/play, the next cycle (when it is closed/destroyed) you get this exception:

E Disable failed. android.media.MediaCodec$CodecException: Error 0xffffffed at android.media.MediaCodec.native_flush(Native Method) at android.media.MediaCodec.flush(MediaCodec.java:2350) at androidx.media3.exoplayer.mediacodec.AsynchronousMediaCodecAdapter.flush(AsynchronousMediaCodecAdapter.java:247) at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.flushCodec(MediaCodecRenderer.java:960) at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.flushOrReleaseCodec(MediaCodecRenderer.java:953) at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.onDisabled(MediaCodecRenderer.java:780) at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.onDisabled(MediaCodecVideoRenderer.java:794) at androidx.media3.exoplayer.BaseRenderer.disable(BaseRenderer.java:220) at androidx.media3.exoplayer.ExoPlayerImplInternal.disableRenderer(ExoPlayerImplInternal.java:1846) at androidx.media3.exoplayer.ExoPlayerImplInternal.resetInternal(ExoPlayerImplInternal.java:1566) at androidx.media3.exoplayer.ExoPlayerImplInternal.stopInternal(ExoPlayerImplInternal.java:1523) at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:577) at android.os.Handler.dispatchMessage(Handler.java:102) at android.os.Looper.loopOnce(Looper.java:214) at android.os.Looper.loop(Looper.java:304) at android.os.HandlerThread.run(HandlerThread.java:67)

E Reset failed. java.lang.IllegalStateException at android.media.MediaCodec.native_stop(Native Method) at android.media.MediaCodec.stop(MediaCodec.java:2300) at androidx.media3.exoplayer.mediacodec.AsynchronousMediaCodecAdapter.release(AsynchronousMediaCodecAdapter.java:268) at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.releaseCodec(MediaCodecRenderer.java:805) at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.onReset(MediaCodecRenderer.java:787) at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.onReset(MediaCodecVideoRenderer.java:804) at androidx.media3.exoplayer.BaseRenderer.reset(BaseRenderer.java:227) at androidx.media3.exoplayer.ExoPlayerImplInternal.resetInternal(ExoPlayerImplInternal.java:1576) at androidx.media3.exoplayer.ExoPlayerImplInternal.stopInternal(ExoPlayerImplInternal.java:1523) at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:577) at android.os.Handler.dispatchMessage(Handler.java:102) at android.os.Looper.loopOnce(Looper.java:214) at android.os.Looper.loop(Looper.java:304) at android.os.HandlerThread.run(HandlerThread.java:67)