MakeAppx fails with 0x8007000e - not enough storage, even when storage is ample

Describe the bug

Builds can create packages locally, but through our Azure DevOps pipeline, the following error is thrown:

_GenerateAppxPackageFile: C:\Users\<user>\.nuget\packages\microsoft.windows.sdk.buildtools\10.0.22000.194\bin\10.0.22000.0\x86\MakeAppx.exe pack /l /h sha256 /f obj\x64\Release\net6.0-windows10.0.19041.0\win10-x64\package.map.txt /o /p Packages<appname>_1.24.0.3027_x64_Test\<appname>_1.24.0.3027_x64.msix

##[error]MakeAppx(0,0): Error : Package creation failed.

##[debug]Processed: ##vso[task.logissue type=Error;sourcepath=MakeAppx;linenumber=0;columnnumber=0;code=;]Package creation failed.

2>MakeAppx : error : Package creation failed. [D:\devops_agent_2\_work\2\s\Source\<apppath>\<appname>\<appname>.csproj]

##[error]MakeAppx(0,0): Error : 0x8007000e - Not enough storage is available to complete this operation.

##[debug]Processed: ##vso[task.logissue type=Error;sourcepath=MakeAppx;linenumber=0;columnnumber=0;code=;]0x8007000e - Not enough storage is available to complete this operation.

2>MakeAppx : error : 0x8007000e - Not enough storage is available to complete this operation.

This is running on a self-hosted agent that has several hundred GB free of disk space, and at no point in the build does the memory reach even 50% utilization. I also get the same error if I change to run on an ms-hosted build agent.

The application is a somewhat large WinUI 3 app. To build the app, we use the following task:

task: VSBuild@1 inputs: solution: 'Source/.sln' msbuildArgs: ${{ parameters.msbuildArgs }} maximumCpuCount: true configuration: 'Release'

with parameters.msbuildArgs being:

'/t:Windows\ /p:AppxPackageDir="Packages" /p:UapAppxPackageBuildMode=SideloadOnly /p:AppxPackageSigningEnabled=false /p:AppxBundlePlatforms="x64" /p:AppxBundle=Never /p:GenerateAppxPackageOnBuild=true /p:PackageCertificateThumbprint=""'

Using Microsoft.WindowsAppSDK version 1.1.3 and Microsoft.Windows.SDK.BuildTools version 10.0.22000.194. Running on Windows Server 2016 Standard, x64, 16 GB RAM.

Steps to reproduce the bug

This would be difficult to produce a minimal repro, since I assume it has to do with the fact that the app is large in size.

As described in the description, an error of "not enough storage" is thrown when running our VSBuild step in our ADO pipeline, despite ample storage space being available.

Expected behavior

Expected behavior would be that no error is thrown and the msix package would be produced successfully.

Screenshots

No response

NuGet package version

No response

Packaging type

Packaged (MSIX)

Windows version

No response

IDE

Visual Studio 2022

Additional context

No response

I think the problem with this is that the error text itself is confusing. The HRESULT 0x8007000E is a Windows API error packaged up in a HRESULT. The Windows API error 0xE is actually ERROR_OUTOFMEMORY. The other thing to note is that the log you are showing there shows that your build is using the x86 version of MakeAppx.exe. Recall that x86 applications have a total of 4GiB of virtual address space. The WoW64 layer cuts off a portion of this meaning that in total, a 32 bit application will have around 3.5GiB of virtual address space available. What's more, due to how processes load libraries, it is impossible to have this as contiguous memory. But there is another issue here, this 3.5GiB is only available if the application declares that it can handle the extra memory. The executable header for MakeAppx.exe doesn't actually include this.

Dump of file makeappx.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               6 number of sections
        E8A81F74 time date stamp
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             102 characteristics
                   Executable
                   32 bit word machine

OPTIONAL HEADER VALUES
             10B magic # (PE32)
           14.28 linker version
           64800 size of code
            5E00 size of initialized data
               0 size of uninitialized data
           643D0 entry point (004643D0)
            1000 base of code
           66000 base of data
          400000 image base (00400000 to 0046EFFF)
            1000 section alignment
             200 file alignment
           10.00 operating system version
           10.00 image version
            6.01 subsystem version
               0 Win32 version
           6F000 size of image
             400 size of headers
           6C93E checksum
               3 subsystem (Windows CUI)
            C140 DLL characteristics
                   Dynamic base
                   NX compatible
                   Control Flow Guard
                   Terminal Server Aware
           40000 size of stack reserve
            2000 size of stack commit
          100000 size of heap reserve
            1000 size of heap commit
               0 loader flags
              10 number of directories
               0 [       0] RVA [size] of Export Directory
           68268 [     104] RVA [size] of Import Directory
           6B000 [     760] RVA [size] of Resource Directory
               0 [       0] RVA [size] of Exception Directory
           6A400 [    21E0] RVA [size] of Certificates Directory
           6C000 [    2CA0] RVA [size] of Base Relocation Directory
           3CD3C [      54] RVA [size] of Debug Directory
               0 [       0] RVA [size] of Architecture Directory
               0 [       0] RVA [size] of Global Pointer Directory
               0 [       0] RVA [size] of Thread Storage Directory
            1000 [      40] RVA [size] of Load Configuration Directory
               0 [       0] RVA [size] of Bound Import Directory
           68000 [     264] RVA [size] of Import Address Table Directory
           656AC [      40] RVA [size] of Delay Import Directory
               0 [       0] RVA [size] of COM Descriptor Directory
               0 [       0] RVA [size] of Reserved Directory

This means that the total virtual address space available to the 32 bit version of MakeAppx.exe is less than 2GiB. Because of where the main executable loads and where libraries loads into the process, you are unlikely to get much more than 1GiB of contiguous virtual address space. I've actually seen times where, due to address space fragmentation, there wasn't even 500MiB of contiguous address space available. So, if the issue here is that MakeAppx.exe is generating the package in memory then it could be that it is running out of virtual address space or even contiguous virtual address space.

The reason why it has no issues working locally seems to be newer versions of Visual Studio default to using the x64 tools. Visual Studio 2022 17.3 shows:

1>MakeAppxArchitecture           = x64
1>MakeAppxExeFullPath            = C:\Users\Darran\.nuget\packages\microsoft.windows.sdk.buildtools\10.0.22000.194\bin\10.0.22000.0\x64\MakeAppx.exe

This means that it has a lot more virtual address space available, and it should have no issue getting 1TiB of contiguous address space, or more. I would honestly suggest that you figure out if it is possible to get the DevOps pipeline to use the x64 tools here. 32 bit tools are known to have this issue, which is why Visual Studio has been moving towards using the 64 bit tools by default.

microsoft / WindowsAppSDK