search

CommunityToolkit / Tooling-Windows-Submodule

Community Toolkit infrastructure for use as a submodule 'tooling' directory in other repositories.
Other
30 stars 10 forks source link

ILT0005 Out of memory when building release mode #9

Open Arlodotexe opened 2 years ago

Arlodotexe commented 2 years ago

Describe the bug

Here goes nothing.

In the CI, we've hit ILT0005, and an out of memory error. Only WinUI 2 failed to build, and no app-affecting code was changed for this CI run.

error : ILT0005: 'C:\Users\runneradmin\.nuget\packages\runtime.win10-x64.microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\x64\ilc\Tools\nutc_driver.exe @"D:\a\Labs-Windows\Labs-
error : Error: NUTC300F:Internal Compiler Error: Native compilation failed due to out of memory error

I've seen this before - I've been having this issue in a project's CI for several months, and I've already spent 2 weeks researching and trying to fix this error.

This is the first time I've seen it happen in Labs, but if this is related, then it's more or less random and only present on mid to low-spec machines. It's not reproducible in a VM and it's going to happen more frequently the more code we put into Labs.

There are some mitigations that we can try anyway, but I'm not expecting it to be that easy.

Steps to reproduce

See https://github.com/CommunityToolkit/Labs-Windows/runs/7550122176?check_suite_focus=true

Expected behavior

A buildable repo should consistently pass CI

Screenshots

image

Code Platform

Windows Build Number

Other Windows Build number

No response

App minimum and target SDK version

Other SDK version

No response

Visual Studio Version

No response

Visual Studio Build Number

No response

Device form factor

No response

Additional context

No response

Help us help you

Yes, but only if others can assist.

michael-hawker commented 2 years ago

For CommunityToolkit/Labs-Windows#211 it was just a general exit code, but it may be the same thing. The guidance from the threads in https://github.com/dotnet/corert/issues/5604 was to turn the 64-bit mode on for .NET Native. I tried that in CommunityToolkit/Labs-Windows#211 but then I think it causes issues with all the MSTest warnings we have about something that's misconfigured there.

Did you want to try just bringing over 92ecbc47988d76bc1057e96fda413a80f9abf6ce in a separate branch and seeing if you can add the missing dependencies (if that's the issue) to the test projects?

"D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj" (default target) (14:6) ->
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Internal Compiler Error [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Type 'System.Xml.XmlDocumentXPathExtensions' from assembly 'System.Xml.XPath.XmlDocument' was not included in compilation, but was referenced in method 'ClientUtilities.FixRelativePathsInRunSettings(XmlDocument, string)'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Method 'ClientUtilities.FixRelativePathsInRunSettings(XmlDocument, string)' will always throw an exception due to the missing method 'XmlDocumentXPathExtensions.SelectSingleNode(XmlNode, string)' in assembly 'System.Xml.XPath.XmlDocument'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Method 'InferRunSettingsHelper.MakeRunsettingsCompatible(string)' will always throw an exception due to the missing method 'XmlDocumentXPathExtensions.CreateNavigator(XmlDocument)' in assembly 'System.Xml.XPath.XmlDocument'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Method 'InferRunSettingsHelper.UpdateRunSettingsWithUserProvidedSwitches(XmlDocument, Architecture, Framework, string)' will always throw an exception due to the missing method 'XmlDocumentXPathExtensions.CreateNavigator(XmlDocument)' in assembly 'System.Xml.XPath.XmlDocument'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Method 'InferRunSettingsHelper.TryGetLegacySettingElements(string, Dictionary<string, string>)' will always throw an exception due to the missing method 'XmlDocumentXPathExtensions.CreateNavigator(XmlDocument)' in assembly 'System.Xml.XPath.XmlDocument'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]

https://github.com/CommunityToolkit/Labs-Windows/runs/7735671450?check_suite_focus=true#step:9:16694

Arlodotexe commented 2 years ago

@michael-hawker In case it helps, we discovered and applied a number of fixes in here (different project) that we could try.

michael-hawker commented 1 year ago

Also see Sergio's comment here: https://github.com/CommunityToolkit/Labs-Windows/pull/290#issuecomment-1290476445

Arlodotexe commented 1 year ago

@michael-hawker This is one of the fixes I intended to try when I got time.

From here, any one of these could help us mitigate the error for now:

    <Use64BitCompiler>true</Use64BitCompiler>
    <PreferredToolArchitecture>x64</PreferredToolArchitecture>

    <SingleThreadNUTC>true</SingleThreadNUTC>
    <UseDotNetNativeToolchain>true</UseDotNetNativeToolchain>
    <EnableGatekeeperAnalysis>true</EnableGatekeeperAnalysis>
    <ShortcutGenericAnalysis>true</ShortcutGenericAnalysis> <!-- Can help stop runaway analysis of generic types and reduce overall generation requirements. -->
    <OutOfProcPDB>true</OutOfProcPDB> <!-- Opt-in to generate PDB's out of process -->

From past experience with this bug, I don't expect any of these to solve the problem permanently, but we can try them.

Arlodotexe commented 1 year ago

In CommunityToolkit/Labs-Windows#211, we managed to make ILT0005 happen on every CI run. After investigating, we found:

[View CI run] ILT0005 happens every time when <Use64BitCompiler> is applied to main equivalent. [View CI run] ILT0005 only happens occasionally when <Use64BitCompiler> is removed.

michael-hawker commented 1 year ago

@Arlodotexe <Use64BitCompiler> was added afterwards to try and resolve the error though...

Arlodotexe commented 1 year ago

@Arlodotexe <Use64BitCompiler> was added afterwards to try and resolve the error though...

ILT0005 is an out of memory error, it could be that using the 64 bit compiler increased the memory used beyond the 4GB limit on 32-bit processes and caused the error to happen more often.

michael-hawker commented 1 year ago

Hitting this a ton on CommunityToolkit/Labs-Windows#405, thought we had some of the flags enabled, but don't see them.

From @Sergio0694:

<Use64BitCompiler>true</Use64BitCompiler> and <OutOfProcPDB>true</OutOfProcPDB> These can help a bit. Need .NET Native 6.2.14 for the later

@Arlodotexe I think these caused other warnings to turn as errors at one point too which we had to resolve, but I'll try pushing a branch with just them enabled in the UWP props file and see what happens.

Arlodotexe commented 1 year ago

A workaround suggested by a member of the UWP Community (see here and here) seems to do what we need in the CI.

There are other variants of ILT0005 that can happen locally (involving PDB write errors), but these are no longer appearing in this project. We'll close this for now, if https://github.com/CommunityToolkit/Labs-Windows/pull/431 passes.