search

RenderHeads / UnityPlugin-AVProVideo

AVPro Video is a multi-platform Unity plugin for advanced video playback
https://www.renderheads.com/products/avpro-video/
235 stars 28 forks source link

Random crash on video loop #1692

Open heritageint opened 10 months ago

heritageint commented 10 months ago

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE

Describe the issue Have a simple text program that loads a short video (10 seconds) from StreamingAssets folder and plays continually on loop - application is crashing (no MediaPlayerEvent or errors) after a random amount of time, sometimes under 10 minutes, sometimes over 90 minutes.

We've added a timecode to the video itself and is always crashing on the last frame of the video so assume it's hanging when trying to loop.

There does look to be a very small memory leak on each loop, but is crashing whilst still well within capacity.

Tested on multiple specs of PC.

Your Setup (please complete the following information):

To Reproduce

  1. Add MediaPlayer and set to loop,
  2. Load video via StreamingAssets
  3. Play Video

Logs If applicable, add error logs to help explain your problem.

Screenshots If applicable, add screenshots to help explain your problem.

Videos If applicable, add a copy of your video or the URL

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE

Instead email the link to us unitysupport@renderheads.com

Chris-RH commented 10 months ago

Is this under Media Foundation, DirectShow or WinRT API?

heritageint commented 10 months ago

Media Foundation

Chris-RH commented 10 months ago

OK, and to confirm, is it D3D12?

heritageint commented 10 months ago

D3D11

Chris-RH commented 10 months ago

Repeatable using the mediaplayer demo scene in Unity editor?

heritageint commented 10 months ago

We've got our sample app running on test (will try the mediaplayer demo after this has completed) - running the original app in the Editor has been running fine for over 3.5 hours, running the original app with a replacement video (removed last two frames to make exactly 10 seconds long) has been running fine for 1.45 hours, so possibly the video file was causing the problem, but will update you if/when they crash.

heritageint commented 10 months ago

Running the MediaPlayer demo (no changes to project, using RH video) on the client PC (Intel Nuc 11 TNKi3 - 8GB RAM) it crashed after approx. 6 hours. - in all versions of our sample app, and the mediaplayer demo, the crash appears to happen at the loop point (end of the video)

heritageint commented 10 months ago

We have ran old applications using Unity 2021.3.23f1 and AVPro Core 2.8.0 on the Intel NUC that crashed previously and no problems (ran over 24 hours) - we can produce a stripped down sample app that just loads a video and loops to test if an issue has been introduced by newer versions of Unity or AVPro, as it doesn't seem to be the PC itself?

Chris-RH commented 10 months ago

Hmmm, that's interesting. After 2.8.0, 2.8.2 had the most changes for Windows and is most likely where the error may have started.

heritageint commented 10 months ago

Do you have a link to V2.8.0 that I could run on test in our sample app, to confirm if that is the issue? We need to find a solution quickly as installing next week.

Chris-RH commented 10 months ago

All the trial version are available here: https://github.com/RenderHeads/UnityPlugin-AVProVideo/releases

heritageint commented 10 months ago

Sample app with V2.8.0 has been running solidly for over 21 hours now, so assume an issue was introduced after this version? Please advise how we can target the Release version of 2.8.0 to allow us to publish our application without the Trial version overlay?

There is occasionally a noticeable lag when the loop occurs in this version, which is obviously preferrable to the software crashing, but perhaps the old version makes multiple attempts to loop (if the first fails) and the later version hangs?

Chris-RH commented 10 months ago

hmm, that's interesting. Thanks for checking. Please can you email us your Unity invoice pdf to unitysupport@renderheads.com, ideally from the same email account that is on the invoice :)

heritageint commented 10 months ago

Hi Chris

I sent our invoice last week as requested - I really need to wrap this up today (being able to build with v 2.8) as we're installing tomorrow morning

Cheers

Bill

Chris-RH commented 10 months ago

email replied to :)

Chris-RH commented 10 months ago

I just wanted to check. is it working again properly with the package that you were sent?

heritageint commented 10 months ago

Yes. Been running solidly.

On Thu, 7 Dec 2023 at 10:07, Chris-RH @.***> wrote:

I just wanted to check. is it working again properly with the package that you were sent?

— Reply to this email directly, view it on GitHub https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/1692#issuecomment-1845049104, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZRWV5MFQBGQQ3K2O5ZWI2TYIGINJAVCNFSM6AAAAAA7P4ZJUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBVGA2DSMJQGQ . You are receiving this because you authored the thread.Message ID: @.***>

heritageint commented 9 months ago

Can you confirm this is a bug with 2.9 and it will be addressed, so we can use the latest version in future projects, as will keep using 2.8 for now?

heritageint commented 8 months ago

Is there an update on whether this was a bug with 2.9 and if/when will be fixed?

Chris-RH commented 8 months ago

We'll update this thread once its fixed and let you know which version that will be :)

heritageint commented 6 months ago

Has there been any progress with this bug? We've just tried again with AVProCore 2.9.3 and even manually looping a video (waiting for FinishedPlaying event and rewinding) it is still causing a sample app (launches and plays a video from StreamingAssets on loop) to crash in under an hour - no errors are logged but crashes after the FinishedPlaying event - we really need a resolution to this as soon as possible as causing issues with released applications.

RichRH commented 6 months ago

Hi @heritageint,

We've recently used AVPro Video (2.9.1 - Media Foundation / Hardware) for one of our client projects which involved a lot of looping (and seeking) and didn't encounter any issues whilst soaking/running live for days.

Would you be able to send your test video to us to see if it makes a difference/we can reproduce it? unitysupport@renderheads.com

Cheers,

TatsuyaPan commented 6 months ago

I encountered the same issue. I initially used version 2.8.0. When running for a long time, the memory occupied would increase rapidly, eventually causing the application to crash. Looking at Unity's Profiler, we found that the increase in memory in the untrack part was the main reason. So I tested 2.8.5/2.9.3 and the latest 3.0.0 version. These versions no longer have the above memory problem, but the application will still crash after a long period of operation. I've found a strong correlation between this and the number of playback loops. When using version 3.0.0, using MediaFoundation playback, approximately 2800-3200 loops can be played; when using WinRT, approximately 1900-2200 loops can be played. Whether I use one MediaPlayer or multiple MediaPlayers, the results are basically the same.

Chris-RH commented 6 months ago

That's interesting to know. Do you get crashes under DirectShow as well?

TatsuyaPan commented 6 months ago

That's interesting to know. Do you get crashes under DirectShow as well?

I tested DirectShow with 2.8.0 before try 2.9.x or 3.0.x, got crashes too.

I tried 1 MediaPlayer last night from 6:30pm to 9:30am. It took 4245 loops without crash. But the memory issue came back, my application ate over 10GB memory, total system memory over 94%. AVPro Video v2.9.3-Trail.

Chris-RH commented 6 months ago

Thank you, that's super helpful :)

RichRH commented 6 months ago

HI @heritageint,

Thanks for sending your test video/scene through - as a quick update I've been soak testing it overnight and haven't yet encountered a crash but have come back to it in a state where it's still running but Media Foundation (Hardware) for some reason isn't returning any new frames.. unless I pause/unpause the editor and it continues as per normal.

I'll try and get to the bottom of it but might take a bit of time given the nature/reproduction requirements of the issue!

Cheers,

heritageint commented 6 months ago

Hi Rich

Just to clarify, I don't believe we've encountered the issue whilst in the Editor, but when built (Windows .exe) it is reliably crashing the application - having just tried it again on a low spec PC (Intel NUC i3 8GB RAM) it only takes about an hour to crash - memory stays static at 30%, GPU at 37%, CPU at 12% - with values staying the same for all apart from the GPU which drops to 0 at the time of the crash.

Cheers

RichRH commented 6 months ago

Ok, thanks for the confirmation @heritageint - I'll try and get a debug build setup to see if I can catch it..

Cheers,

RichRH commented 6 months ago

Hi @heritageint,

Quick update - it's been running all night as a debug build in soak test without issue (memory is also fine - ~60MB). I am running latest code though (equivalent of the unreleased 2.9.4/newly released 3.0.x) - just wanted to check if you've tried v3? (Trial I think should be fine)

Cheers,

TatsuyaPan commented 6 months ago

Hi @heritageint,

Quick update - it's been running all night as a debug build in soak test without issue (memory is also fine - ~60MB). I am running latest code though (equivalent of the unreleased 2.9.4/newly released 3.0.x) - just wanted to check if you've tried v3? (Trial I think should be fine)

Cheers,

Hi ,thanks for the work. I tested the lasted trail version which released last Friday. It caused memory issue as well. By the way, the "loop" I said is the method pair "OpenMedia" and "CloseMedia"—— PlayMedia then CloseMedia . I tested the field "loop" all night with a video which length is 10 seconds , and the memory used was healthy. Maybe the open/close operation trigger the issue faster? Sorry for my bad English. And hope my test can help.

heritageint commented 6 months ago

Hi Rich Just tried V3 and still crashed, however with this version there are some debug errors - it reported "Direct3D: detected that vsync is broken (it does not limit frame rate properly). Delta time will now be calculated using cpu-side time stampling until the game window is moved.". I've attached our debug file, the error is towards the end. I also noticed that when it crashed with V3, the application still kept running (previous versions get a Windows stopped responding popup) and just showed a blue screen (default camera background) - was still using GPU whereas before dropped to 0. Cheers Bill Player.log

heritageint commented 5 months ago

Has there been any update with the loop bug - were you able to recreate it? We urgently need a fix for a number of installs.

Chris-RH commented 5 months ago

This is being looked into. We're just struggling to reproduce it at the moment. We're going to try and run some more tests on a lower spec machine to see if we can force it.

heritageint commented 4 months ago

Has any progress been made with this issue? We are now in the process of having to remove AVPro from a number of our applications running on sites and replace with the standard Unity Video Player which is very disappointing given it's 6 months since the bug was reported.

Chris-RH commented 4 months ago

We think its due to an event not being fired, but we are still digging into it. We've not been able to recreate an actual crash yet, but have found something similar. We do a lot of soak testing ranging from overnight to several weeks.

RichRH commented 4 months ago

Hi @heritageint,

To follow on from Chris' post - apologies this is taking so long to address. We're a small team and have recently been slammed with work, but a lot of that work has relied on us using AVPro Video (Windows) for playback in high-stress (live) environments so we've been heavily soak testing with this particular issue in mind.

The reality is we've not encountered a single issue in our projects, which is frustrating as we've half hoped to see similar! We do now have a bit of a gap between projects so are looking deeper into it.

A few notes so far:

Further questions:

Cheers,

heritageint commented 3 months ago

Hi Rich

Thanks for continuing to investigate.

We added the FinishedPlaying event to allow us to manually add Debug logging (which led us to confirming the issue was with the loop, not mid-sequence). We had the same issue using the looping flag.

We usually use MediaFoundation, but can try with DirectShow and WinRT and report back.

On your first point with the playback stopping at the loop point, we noticed this behavior in V3 (demo), whereas in V2.8/2.9 it would crash, so perhaps some elements have been fixed in the latest version. We will add the Play command after the Rewind command and run that on test using V3.

Cheers

RichRH commented 3 months ago

Hi @heritageint,

Thanks for the feedback.

An update from my side - I've written a minimal test case to try and reproduce the looping issue (outside of Unity) using a shortened version of your test video (1s), MF, basic software playback and both manual and auto-looping. I could get it to freeze (stop delivering new frames) relatively quickly, but it would be triggered by some very odd interactions.. such as clicking through emails in Outlook! (I suspect triggering a CPU/GPU delay that impacted MF somehow).

Anyhow, after digging deeper and in combination with a few other tweaks, I appear to have found the main cause(s) in my test. Overnight I ran my fixes on 3 machines (including the low spec machine that was predominantly exhibiting the issue) that are now all still happily running after 55000+ iterations.

I'm working towards getting these changes into a new build (although a new project is also ramping up) but in the meantime you may find some success in removing the following in 'WindowsMediaPlayer.cs' -> 'IsFinished()' with your current setup:

if (!result)
{
    // This fixes a bug in Media Foundation where in some rare cases Native.IsFinished() returns false
    result = (GetCurrentTime() > GetDuration());
}

I suspect the crash you were seeing may be a different issue, but a consequence of the freeze - I'm still yet to experience it but will keep it in mind when reintegrating these changes.

Cheers,

RichRH commented 3 months ago

Quick update on this - 6 tests on 3 different machines have been running all weekend and we're now up to 300000+ iterations which gives me confidence that the fixes are good.

I did find another chunk of code that needed removing in combination with the above change to avoid finished messages sometimes not being reported - in 'MediaPlayer_Events.cs' -> 'CheckAndClearStartedAndFinishedEvents()' comment out:

#if UNITY_EDITOR_WIN || (!UNITY_EDITOR && (UNITY_STANDALONE_WIN || UNITY_WSA))
reset = false;
if (_infoInterface.HasVideo())
{
    // Some streaming HLS/Dash content don't provide a frame rate
    if (_infoInterface.GetVideoFrameRate() > 0f)
    {
        // Don't reset if within a frame of the end of the video, important for time > duration workaround
        float secondsPerFrame = 1f / _infoInterface.GetVideoFrameRate();
        if (_infoInterface.GetDuration() - _controlInterface.GetCurrentTime() > secondsPerFrame)
        {
            reset = true;
        }
    }
    else
    {
        // Just check if we're not beyond the duration
        if (_controlInterface.GetCurrentTime() < _infoInterface.GetDuration())
        {
            reset = true;
        }
    }
}
else
{
    // For audio only media just check if we're not beyond the duration
    if (_controlInterface.GetCurrentTime() < _infoInterface.GetDuration())
    {
        reset = true;
    }
}
#endif

I'm hoping to get these fixes out soon.

Cheers,

heritageint commented 3 months ago

Hi Rich

Thanks for this.

Just to double check, should this help with version 2.9 or would it only work as a fix for V3, as we're still running V2.9 until a fix was found?

Cheers

Bill

RichRH commented 3 months ago

Hi Bill,

I would expect these changes to impact 2.9 (and before - not sure how far though) as much as v3 as it's very similar code at this level.

I've continued running soaks and everything seems good so aim to get these changes in for a build on Friday, all being well. It could well be there are other parts to this issue but I'm aiming to make sure fundamentally playback is solid before addressing anything further, if there are indeed any outstanding issues relating to this.

Cheers,

RichRH commented 3 months ago

Hi @heritageint,

3.0.4 has just been released featuring some of the above fixes - would be great to know if it helps?

Cheers,

heritageint commented 3 months ago

Hi Rich

We ran our SW with 2.9 and the code commented out you suggested and this had no effect, crashed relatively quickly. We've downloaded the trial version of 3.0.4. and have that running now and will report back once it's been ran over the weekend.

Cheers

Bill

RichRH commented 3 months ago

Thanks for the update Bill (@heritageint) - what do you mean by crashed here?

One further note - I added a 'Renderer Sync' flag for Media Foundation in 3.0.4 that defaults to on.. if you're still getting crashes it would be good to know if disabling this flag has any impact.

Cheers,

heritageint commented 3 months ago

Crash = it fails to loop the video (so display is frozen on last frame) then when touch/click the screen the SW crashes with a Not Responding OS popup message.

heritageint commented 3 months ago

We've been running the demo of 3.0.4 continually over the weekend with no crashes which is great news - I assume this means we will have to upgrade to V3 or can the fix be applied to V2? As mentioned above commenting out the code you suggested had no effect.

RichRH commented 3 months ago

That is good news @heritageint! I left 6 test builds running here over the weekend too, and all are still good.

I think upgrading to v3 is the best option as it would be tricky to retrofit these changes into v2 given it's an amalgmation of multiple fixes and the two paths have already become quite divergent.

Cheers,

heritageint commented 3 months ago

Unfortunately had a crash with V3.0.5 yesterday running a new project with a simple video loop (left looping all day, crashed after about 8 hours).
In this project we did use the Loop checkbox, so are trying a version now with the existing code waiting for the FinishedPlaying event and then rewinding - will report back after tests.

RichRH commented 3 months ago

Hi @heritageint,

Ah, that's frustrating - I'd be surprised if it's a regression but immediate thoughts/questions:

Cheers,