Low FPS and scene stuttering with high resolution H.264 and H.265 files with alpha packing

michelepanegrossi commented 2 years ago

Describe the issue A clear and concise description of what the issue is.

Videos stutter during playback, FPS drops as low as 18FPS, scene stutters and general inconsistencies. Videos don't loop seamlessly. Scene stutters badly on multiple videos rewind.

Using HDRP and Display IMGUI component

Your Setup (please complete the following information):

Unity version: 2021.2.9f1
AVPro Video version (number and edition (trial/core/ultra/enterprise)): 2.5.1 core version just updated to try solve the issue but doesn't seem to fix it.
Operating system version: Windows 10 21H1
Device model: Razer laptop with Intel i7-10875H 2.30 GHz and RTX 2080 super 8GB, Desktop with AMD Ryzen 5900 12 cores and Nvidia RTX 3080. Smoother on the second but problem persists.
Video specs (resolution, frame-rate, codec, file size):

Tried with H.264 and H.265 videos, 60 FPS, some 4320 x 1652 resolution, some 4320 x 1600 resolution. Videos have alpha packing to the right. Different sizes and data rates:

48.79 Mbit/s @ 178MB size, 29 seconds duration, H.264 14180 Kbit/s @ 7.1 MB (this one loops), 4 seconds duration H.265 with fastdecode and faststart flags 9.79 Mbit/s @ 14MB, H264

To Reproduce

Install unity HDRP, set game window resolution to 2160 x 3840 portrait
Add a 'shower like' VFX graph particle system to the scene, just wide enough to fill the camera view. 10000 particles with capacity of 1000000. Use collision with depth buffer.
Add a humanoid avatar controlled in real time with an azure kinect depth camera.
Set videos to play as Display IMGUI components. Switch instantly between videos by changing the opacity of the IMGUI components via script and playing them.

Logs If applicable, add error logs to help explain your problem.

Screenshots If applicable, add screenshots to help explain your problem.

Videos If applicable, add a copy of your video or the URL

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE

Instead email the link to us unitysupport@renderheads.com

I tried encoding the looping video with H265 and faststart and fastdecode as explained in the encoding notes. It helps but does not solve the issue. an H264 version of the same file doesn't loop seamlessly at all.

When I try and switch between videos by changing the opacity

All videos are set to load on start, but not play (except for the looping video that also plays on start). All videos have only 'hardware decoding' checked and nothing else in the windows options. I tried to follow the advice from issue #1111 but didn't get huge improvements.

What would be the best way of encoding the files for this use case? Is the resolution too high? I looked at the microsoft Media Foundation specifications and the resolution seems to exceed the 4096 x 2304 limit? But does it matter with the alpha packing?

Is there anything else I am doing wrong? Or Am I just doing too much in my scene? I tried to optimise the HDRP settings by switching off what I don't use.

Could there be conflict between the VFX graph and AVPRO? I am also using some GPU events when particles die colliding with the depth buffer and spawn other particles.

Particles are lit quads.

Any help would be appreciated.

Michele

Chris-RH commented 2 years ago

Hi @michelepanegrossi It does look like a lot for your GPU. Are all the drivers up to date? Could you try watching the decoder while its running to see what is happening there? Task Manager -> Performance -> GPU Decode Let us see how that goes and then we can delve into the rest of your questions further.

michelepanegrossi commented 2 years ago

Thanks @Chris-RH I have some updates. I experienced a massive performance boost by doing the following: update to the latest version of the AvPro Core asset. Reduce the resolutions of the videos to 4096 1652 (the limit stated in the documentation) down from 4320 1652 and encoding all the videos as H.264 without transcoding with additional flags from ffmpeg.

To answer your question, the video decode sits at about 25% for a single video playback both running in editor and fullscreen build. it goes up to around 50% with 2x videos playing at the same time.

The GPU utilization however is a lot higher, sitting around 50% in build and around 85% in the editor.

In my build, there is only one moment now when unity freezes, which is when I try and rewind all the videos in a for cycle.

In that moment I can see the GPU hitting 100% both in build and editor.

What can I do to mitigate this? Try rewinding the videos in a coroutine adding yield return null inside the for cycle?

Here is the code I am using:

public void ResetAllVideos()
{
        foreach(MediaPlayer m in _allMediaPlayers)
        {
            m.Stop();
            m.Rewind(true);
        }

        _handGestureVideo.Rewind(true);

        _allMediaPlayers[0].Play();
}

Michele

Chris-RH commented 2 years ago

How many videos are you trying to play/rewind at the same time?

michelepanegrossi commented 2 years ago

I have an array of Media Players that stores 8 references (each Media Player has a Media Reference assigned, which links to a video with transparency, left right packing).All videos are auto loaded on start, apart from one which also plays on start and loops. Each time a switch happens I callDisplayIMGUI.setColor(1,1,1,0)On the current video and callDisplayIMGUI.setColor(1,1,1,1) on the video to play next.I then call Stop() and Play() on the current and next video.This seems to work.I'm always playing one video at any one time, performing several switches as described above.This seems to work well at the moment on the desktop machine with Nvidia RTX 3080 GPU (on the razer laptop video playback is really stuttery but I don't have the log from that machine, I will have it next week).When I call Stop() and Rewind() on each video in the for cycle, I'm calling it on 8 videos.On 30 May 2022 10:03, Chris-RH @.***> wrote: How many videos are you trying to play/rewind at the same time?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.> On 30 May 2022 10:03, Chris-RH @.> wrote: How many videos are you trying to play/rewind at the same time?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

Chris-RH commented 2 years ago

What's it like if you just do a couple of videos?
Can you rewind each video after it has stopped playing rather than all at once?
Ideally, the best solution would be to combine them into one mega video and potentially seek to different points.

michelepanegrossi commented 2 years ago

What's it like if you just do a couple of videos?

Can you rewind each video after it has stopped playing rather than all at once?

I will have access to the machine on Monday and I'll.be able to test the above. Which one would be the most efficient Rewind function to call? Media player.Rewind(bool pause), MediaPlayer.RewindPrerollPause() or MediaPalyer.Control.Rewind()? And what is the difference exactly between them?

Ideally, the best solution would be to combine them into one mega video and potentially seek to different points.

Actually this is what I tried originally but the performance wasn't good so I started looking for an alternative solution. If I were to try again, which would be the most efficient seeking function to use? Seek, SeekFast, SeekToFrame or SeekToFrameRelative? Or Media player.SeekToLiveTime? And what does this last one do exactly?

Also, what would be the best way to encode the video? From the encoding notes I would go with:

ffmpeg -hide_banner -y -i input.mp4 -pix_fmt yuv420p -c:v libx264 -crf 18 -tune fastdecode -x264-params "keyint=1" output-h264.mp4

But I'm not clear on whether this would give a fast and accurate way of seeking in relation to the different functions available.

Also what would be the right sequence? Pause video then seek then Play? Or Play then seek?

When I did my test I didn't follow the instructions on this link https://www.renderheads.com/content/docs/AVProVideo/articles/feature-seeking-playbackrate.html

But I did encode the videos with keyframe distance 1.

AndrewRH commented 2 years ago

What's it like if you just do a couple of videos?

Can you rewind each video after it has stopped playing rather than all at once?

I will have access to the machine on Monday and I'll.be able to test the above. Which one would be the most efficient Rewind function to call? Media player.Rewind(bool pause), MediaPlayer.RewindPrerollPause() or MediaPalyer.Control.Rewind()? And what is the difference exactly between them?

All the rewind functions are the same in terms of efficiency. When you rewind a paused video, you're asking the player to decode the first frame of the video. So it's just a function of how much effort it takes to decode that first frame, which is based on the encoding. This is also taking up GPU decode resources, which is why if you try to rewind 8 videos at once, you're using up 8 GPU decode instances which for such high resolution videos is beyond the GPU capabilities, so it must queue them and this leads to the rewinds taking longer to complete.

MediaPlayer.Rewind(bool pause) will just rewind the video and optionally pause it too. MediaPlayer.Control.Rewind() is the same as the above, it's just the internal method. MediaPlayer.RewindPrerollPause() is used for Smooth Video buffered frames is used and it will rewind and start decoding the first 7 frames into a buffer to help with smoother playback. This is the most expensive, but it happens over several frames.

Ideally, the best solution would be to combine them into one mega video and potentially seek to different points.

Actually this is what I tried originally but the performance wasn't good so I started looking for an alternative solution. If I were to try again, which would be the most efficient seeking function to use? Seek, SeekFast, SeekToFrame or SeekToFrameRelative? Or Media player.SeekToLiveTime? And what does this last one do exactly?

SeekFast() will seek to the nearest keyframe. This is the fastest method to use, however it may not be very accurate in terms of where it seeks to, as it is effectively snapping to the nearest keyframe time instead of the exact time specified. If you add more keyframes to your encoding then it will become more accurate (at the expensive of file size). It is also possible to force keyframes at certain parts of your video, but this is quite advanced encoding and can be very fiddly.

SeekToFrame() is just a wrapper for Seek(double time) which tried to convert from time into frames.

SeekToFrameRelative() is the same as the above, but it jumps forward or back relative to the current position instead of using an absolute time.

SeekToLiveTime() is only useful for live streams and allows you to seek back in time relative to the head of the live stream.

I'm not sure why performance would be bad when creating a single large video. In what way was it bad? If you want fast seeking then the fastest will be if you video is all keyframes. Otherwise you can also simplify your video encoding and perhaps adding more keyframes as a middle ground.

Also, what would be the best way to encode the video? From the encoding notes I would go with:

ffmpeg -hide_banner -y -i input.mp4 -pix_fmt yuv420p -c:v libx264 -crf 18 -tune fastdecode -x264-params "keyint=1" output-h264.mp4

But I'm not clear on whether this would give a fast and accurate way of seeking in relation to the different functions available.

Yes, this would give you the fastest seeking, and it would be accurate.

Also what would be the right sequence? Pause video then seek then Play? Or Play then seek?

It depends what you want to do. You can load the video with "auto play" disabled. Then you can Seek and Play.

When I did my test I didn't follow the instructions on this link https://www.renderheads.com/content/docs/AVProVideo/articles/feature-seeking-playbackrate.html

But I did encode the videos with keyframe distance 1.

If you're still having seek issues feel free to email us a sample of your video to unitysupport@renderheads.com with the subject "Github #1190".

Thanks,

michelepanegrossi commented 2 years ago

Hi @AndrewRH and thank you so much for the details on the Rewind() and Seek() functions.

To answer points 1. and 2. from the previous message:

If I Rewind only a couple of videos in the for cycle the freeze doesn't occur. As soon as I add more videos it comes back. I didn't spend time checking what is the threshold at which the problem occurs (how many videos rewinded together in the for cycle).
Yes I can but it appears to cause some stuttering to the other video that starts playing.

The solution that seems to work best is to use a coroutine, such as this one:

IEnumerator ResetAllVideos()
{
        foreach(MediaPlayer m in _allMediaPlayers)
        {
            m.Rewind(true);
            yield return null;
        }

        _handGestureVideo.Rewind(true);

        _allMediaPlayers[0].Play();
}

this way the freeze doesn't seem to occur.

To answer your question about seeking, the problem I was having was that during seeking the unity scene would freeze / stutter momentarily. Maybe I wasn't waiting properly for the seeking to finish before playing the video. I gave up fairly quickly and maybe it would work with the reduced resolution videos I am using now (4096 x 1652) .

Unfortunately I don't have the capacity to revert to seeking for this project but I will definitely try again in future.

AndrewRH commented 2 years ago

@michelepanegrossi We've just released 2.5.2 which has some performance improvement so you might want to give it a try

michelepanegrossi commented 2 years ago

Thanks @AndrewRH I will try the new version as soon as possible as the stuttering on rewinding the videos still seems to occur at times. It doesn't seem to be consistent.

Thanks so much for your assistance with this matter

Chris-RH commented 2 years ago

Hi @michelepanegrossi Have you had a chance to try the new version? We're hoping that you will see some improvement.

RenderHeads / UnityPlugin-AVProVideo

Low FPS and scene stuttering with high resolution H.264 and H.265 files with alpha packing #1190

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE