search

RenderHeads / UnityPlugin-AVProVideo

AVPro Video is a multi-platform Unity plugin for advanced video playback
https://www.renderheads.com/products/avpro-video/
235 stars 28 forks source link

Huge memory leak #1736

Closed Pasmich closed 2 weeks ago

Pasmich commented 8 months ago

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE

Describe the issue

Context:

Issue:

Your Setup (please complete the following information):

To Reproduce

  1. Play a lot of different videos one after the other on multiple players in the same scene, either with Media Foundation or Direct Show
  2. See mem usage build up

Logs If applicable, add error logs to help explain your problem.

Screenshots If applicable, add screenshots to help explain your problem.

Videos If applicable, add a copy of your video or the URL

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE

Instead email the link to us unitysupport@renderheads.com

Chris-RH commented 8 months ago
  1. How are you loading the media? Are you opening a new media player for each video and closing it when the video ends or are you using the same media player and changing the path at runtime?
  2. Is the media stored locally or streaming?
  3. Have you tested in a new project using just AVPro playlist demo, with your videos set up in the playlist?
  4. How many videos are you playing at the same time?
  5. How long does it take for the memory leak to cause it to crash?
  6. If you look in Task Manager, what are your GPU/CPU/RAM levels like while this is occurring?
  7. How big is the memory leak?
  8. Does it slowly keep increasing over a time or does it rise to a certain point and then stop?
  9. Have you tried garbage collection?
Pasmich commented 8 months ago
  1. We are using reusing the same MediaPlayer components. On every video open, we do an OpenMedia() with a new MediaPath with a string generated at runtime. We are also not using MediaReference assets, we are simply feeding the paths directly.
  2. Synced from a Google shared drive with local copy enabled.
  3. We haven't yet tried using a demo scene, only a homemade test scene.
  4. We have a maximum of 12 videos at once in the "worst" case, on average we have about half of that (it depends on user interaction with the scene).
  5. On our staging machine with 64Gb of RAM, it took about 12 hours (we launched the app at end of day from 5pm and it crashed at about 4 AM)
  6. We would need to test again and look directly. Since we were testing builds overnight, I relied mostly on Unity's memory snapshot package to monitor memory footprints as opposed to the Task Manager. The only time we used Task manager, the app was left to run over the weekend and on the next Monday, the app had crashed (frozen) and the ram was full.
  7. From our observations with our test scene, using the Memory Profiler package from Unity to take snapshots, we saw an increase of about 1 Gb of "Untracked"/native memory on RAM every hour. At about 100+ Gb taken (taking into account potential virtual memory use) the app would crash.
  8. The increase is consistent, and can be seen quickly through taking memory snapshots after open/closing videos in a loop. Using the WinRT API, there is no leak and after many hours the memory stays consistent at around 0.8 Gb total.
  9. Yes, we tried testing by calling, on video close, everything that appeared to be at our disposal, namely: MediaPlayer.ForceDispose(), Resources.UnloadUnusedAssets(), and GC.Collect(). The unload logged 0 assets freed and so did not seem to do anything, and since the leak happens in untracked memory, I doubt the GC collection could save us... Using these calls displayed no change.
Pasmich commented 8 months ago

Any news on this?

Chris-RH commented 8 months ago

This is being looked into :)

Pasmich commented 8 months ago

Great, thanx!

Pasmich commented 7 months ago

Hi there, any news on this? It's a bit sad we had to revert to using the default unity video player in our project. I have a new project coming on where i would like to be able to use AvPro for it's HAP/Notch support, but i don't feel comfortable at the moment.

unity-dev-01010101 commented 7 months ago

@Pasmich As an alternative for Hap you can try this. I use it from time to time and never had any memory leaking issues with it (also looks like they fixed something on that side recently from the changelog). And always stable playback. Hap R can be used instead of Notch LC with this encoder - that one applies even if you decide to go with AVPro path I guess

RichRH commented 6 months ago

Hi @Pasmich,

Quick update - we've recently used AVPro Video (2.9.1 - Media Foundation / Hardware) for one of our client projects and in the process did some leak checking at native level, but barring a couple of very small leaks (fixed for 2.9.4 onwards), couldn't spot anything major in system or GPU memory. It may be a) they've already been addressed, or b) occuring based on your particular use-case at C# level.

As you have a test scene demonstrating the exact issue, are you able to share it with us per chance? unitysupport@renderheads.com

Cheers,

wybo commented 6 months ago

I've encountered what is probably the same issue. Using 2.9.1 on Windows, playing short (few seconds) videos in quick succession for a few hours, and AVPro will balloon to eat 17+ GB of Ram and crash.

1) 2 Media players in PlayListMediaPlayer setup. Using OpenMedia. With SwapPlayers modified to public, to be able to swap programmatically myself at a set time.

2) Locally, on SSD

3) Haven't.

4) Regularly 2 (never more) as the SwapOver happens before one finishes.

5) Will depend on RAM. But several GBs growth per hour (it crashes out after about 4 on a 24GB machine with other things running). Making the above reported 1 GB to definitely be in the ballpark.

6) It inexorably grows, sometimes stepping back a little. But it keeps climbing. Starting at around 1300mb for the app, going up in a semi linear fashion.

7) Several GBs per hour at least. Only RAM.

8) Goes beyond 17GB at least.

9) Not tried. But profiling showed it to be in untracked memory.

Details: Unity: 2023.1.9 AVPro: 2.9.1 Core Videos: H264 - MPEG-4 AVC (part 10) (avc1), resolution 1920x1072, 60fps OS: Windows 10 Pro 22H2 Hardware: AMD Ryzen Threadripper, 8 Cores & Nvidea RTX 2080

Chris-RH commented 6 months ago

Could you test the trial version of 3.0.0RC2 and see if you are still getting the memory leaks please? Another user has found that issues may increase with the number of playback loops. 2800-3000 in MediaFoundation and 1900-2200 in WinRT. Can you see if this causes issues for you, especially in terms of memory.

If there are still issue after all this, please could you send us a test project to unitysupport@renderheads.com

Chris-RH commented 6 months ago

Could you test the trial version of 3.0.0 and see if you are still getting the memory leaks please? Another user has found that issues may increase with the number of playback loops. 2800-3000 in MediaFoundation and 1900-2200 in WinRT. Can you see if this causes issues for you, especially in terms of memory.

If there are still issue after all this, please could you send us a test project to unitysupport@renderheads.com

wybo commented 5 months ago

I've upgraded to 3.0.0 before the weekend. The memory use still increases at about a gb an hour.

Am running some more tests right now.

Do these playback loops have to be loops for the symptoms the other user found? As I'm not doing any loops. They're different 6 seconds long videos virtually all the time, selected out of a pool of between 40ish and hundreds. With new MediaPaths from files on disk for every play.

They're opened with: playListPlayer.NextPlayer.OpenMedia(new MediaPath(((VideoItem) item).visualPath, MediaPathType.AbsolutePathOrURL), autoPlay: false);

Then when pre-loaded, at an exact time (on the beat of music) the players are swapped: playListPlayer.NextPlayer.Play(); playListPlayer.SwapPlayers();

(had to make some private things public to make this work)

The logs indicate it's using MediaFoundation. With 10 different videos per minute, 2800 would be 280 minutes, or 3 hours and 20 minutes, which is later than the onset time of the gradual increase, but might match well with the escalation of the performance issues 3-4 hours in on version 2.9X. Am testing 3.0 now on a more typical users machine with 8gb of ram.

What I did find however, on an earlier test of 3.0, is that upon setting the GameObject that the Display uGUI is rendering to, to inactive, with gameObject.SetActive(false) for 10 minutes or so, much of the memory clears.

Chris-RH commented 5 months ago

Thanks I think looping one video is just a very easy way to test, as its exactly the same video each time, so its just the number of times its played that increases. It reduces variables.

wybo commented 5 months ago

Have ran a few test-runs now on the side, still without profiler. Just to observe what happens after the upgrade. And it seems the gradual hourly increase only goes so far, and goes back down when it needs to. I do run the garbage collector manually every 40th play, but that doesn't seem to be what gets it down, nor the inactivation of the GameObject, as without that it sometimes goes down again as well. So it's now likely non-leaked garbage hanging around. If you have any tips on the optimal GC setup, while still maintaining performance, please let me know.

The main issue however is the 2800-3000 plays MediaFoundation crash. Every time it crashed. Once it lasted towards 5 hours at one play per 10 secs. But usually around the 3 and a half hour mark as you indicated the other user had happen.

RichRH commented 3 months ago

Hi @wybo,

We've just released 3.0.4 which fixed some issues with looping/crashes as seen in https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/1692 - are you able to see whether this also addresses your crash issues?

Cheers,

wybo commented 3 months ago

Updating now. Will be running it all night this evening. Will let you know tomorrow!

Chris-RH commented 3 months ago

Have you had any luck with that @wybo ? :)

wybo commented 3 months ago

It kept running during the night. Which is an improvement over crashing. Though it still used nearly all of my 48gb of memory.

Now I know that because of C#s lazy garbage collection it postpones this until it hits the memory limit. Once I started other applications it freed up about half of it. Still have to test it with a memory profiler, but have been very busy with my product release. Hope to test that this week.

Chris-RH commented 2 months ago

Thanks, let us know how you get on please.

Ste-RH commented 3 weeks ago

We will be closing this issue soon due to lack of activity. If possible, please update us @wybo

RichRH commented 2 weeks ago

Quick update - I've located a memory leak (in Unity) that I'm investigating and may well fix the above. I'm also going to close this ticket in favour of the more recent (but really a duplicate) one here: https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/1977

Cheers,

wybo commented 1 week ago

Just did another round of tests over the last 2 days, and it is still happening. Results in a mem-leak & crash within 2 hours on a 8gb ram machine. Looking forward to the fix in #1977!