SRombauts / UEGitPlugin

Unreal Engine 5 Git LFS 2 Source Control Plugin (beta)
http://srombauts.github.io/UEGitPlugin
MIT License
828 stars 164 forks source link

Out memory error 4.25 with DX12 RHI #139

Closed duplexsystem closed 2 years ago

duplexsystem commented 4 years ago

Withe the latest version form source control I get an out of video memory error when trying to save. I have 11gb of video memory and the project is 1gb so there is no conceivable way it could run out. (don't know why it would even be using VRAM)

Carbon-NateBowman commented 4 years ago

To add some info to this issue.

image

Issue is easily reproducible by me and my team when checking out any asset config/texture/level etc with RHI set to DirectX12.

It only occurs for me and my team when using dx12, past dx11 projects work fine.

After opening a working existing dx11 project, switching RHI to DX12, and restarting the editor, the issue occurs upon first checkout.

The issue occurs with release 2.16-beta and still occurs after https://github.com/SRombauts/UE4GitPlugin/commit/214af6df6adbd133f616b6191b090012cafb7f3c

@SRombauts I cannot find a workaround for this, and as my team rely on an LFS locking workflow we are a bit stuck until it is resolved.

duplexsystem commented 4 years ago

@Carbon-NateBowman after getting this with another plugin I determined it was a UE 4.25 bug. You can attempt to verify this is you want. But I do no know if it can be fixed plugin side or has to be fixed engine side.

SRombauts commented 4 years ago

Thanks for reporting this and updating me. The question is, is this happening without Source Control enabled ?

Carbon-NateBowman commented 4 years ago

Thanks for reporting this and updating me. The question is, is this happening without Source Control enabled ?

It does not appear to happen without source control enabled. For me, DX12, no source control, at least 5hrs a day since 4.25 release, not happened once.

duplexsystem commented 4 years ago

I've had it happen with the megascans plug-in (another plugin which would be doing similar UE side operations) only starting with 4.25, with dx12 which is far to specific to be unrelated.

Carbon-NateBowman commented 4 years ago

I've had it happen with the megascans plug-in (another plugin which would be doing similar UE side operations) only starting with 4.25, with dx12 which is far to specific to be unrelated.

Oh, that's a good point, when 4.25 came out, I also had that (but straight ctd, no error) when importing large megascans assets with their first 4.25 plugin versions. It was resolved after a few revisions. As a test, I just imported a few 8k surfaces and high poly models (with all lods) without issue.

Carbon-NateBowman commented 4 years ago

UE 4.24.3 with 2.15-beta RHI set to DX12, spikes on checkout but not fatal for small numbers of assets. However, checking out multiple items causes the same kind of spike and eventually a crash, just much slower.

Checking out 25 items. image

duplexsystem commented 4 years ago

Which makes me think it's not a bug with update to 4.25 but a bug or change with ue that requires a plug-in side work around or change.

crevetor commented 4 years ago

I'm having the same issue :(

SRombauts commented 4 years ago

Sorry guys, I am having more things to deal with on my day job, and at the same time more things to do with the family at home. Thanks a lot @crevetor for fixing this for us all

Carbon-NateBowman commented 4 years ago

Better results in 4.25 but unfortunately not fixed completely.

see https://github.com/SRombauts/UE4GitPlugin/pull/142#issuecomment-658848702

SRombauts commented 2 years ago

Still present with UE5.0 & pre-UE5.1 built from sources

see https://github.com/SRombauts/UE4PlasticPlugin/issues/86

I have started reproducing the issue in Unreal without any Source Control operation, with a simple infinite loop in the Provider, meaning it's purely a render thread issue (and I can see how other progress bars in the Editor code seem to handle this)

2022-08-17 (2)
SRombauts commented 2 years ago

A similar bug https://issues.unrealengine.com/issue/UE-93603 Directx 12 RHI can run out of video memory on long process (anims import)

has been fixed by adding End & BeginFrame on the RHI thread https://github.com/EpicGames/UnrealEngine/commit/4cbce19c2848723ccc01a4fba3133d663d64d4ae

SRombauts commented 2 years ago

And the official issue would be https://issues.unrealengine.com/issue/UE-100291 "Out of video memory" crash when attempting to save many assets with source control enabled.

A very recent one even mention the workaround I implemented in the plastic plugin! https://issues.unrealengine.com/issue/UE-160299 GPU memory leak during large source control operations:

The crash has been seen in Git and SVN, and can be avoided by removing FScopedSourceControlProgress::Tick. This will prevent Slate from being ticked during the operation, but will prevent the throbber widget from animating.

SRombauts commented 2 years ago

PR to Epic Games :) https://github.com/EpicGames/UnrealEngine/pull/9492 [UE-100291][UE-160299] Fix source control progress bar crashing out of video memory in case of long-running operation #9492

SRombauts commented 2 years ago

The proper fix has been submitted to ue5-main https://github.com/EpicGames/UnrealEngine/commit/67dacf2baaafe0d9b6fb5eeb5b179fb5b871a4bf

It will be part of UE5.1 release :)