Closed robotdjman closed 8 months ago
Any luck with this? I'm agetting the exact same error on 5.3.2. I was able to run the build command successfully directly from my Win11 desktop but running the same command in a VM fails with the same error you get.
For other instances of that error code (3221225781 == -1073741515 == 0xC0000135 == STATUS_DLL_NOT_FOUND
), it's been missing redistributable or similar, so the process couldn't load a DLL it needed to start.
Weird that happened for a binary we (presumably) compiled earlier in the build though. Maybe it depends on a DLL that came in binary form from gitdeps, and that is the one which needs another redistributable installed.
If you're able to reproduce this in a VM, i.e. with a UI, I expected you'd get a popup saying which DLL was missing. Otherwise, you could use a tool like https://pypi.org/project/dll-diagnostics/ or https://github.com/lucasg/Dependencies on UnrealPak.exe
to see which DLLs are required, and then we can work out what redistributable they come from and add them to the Dockerfile.
From what I can tell C:\UnrealEngine\FeaturePacks\TP_HandheldARBP.upack doesn't actually exist on the container where this fails. I see it when i attach to the container which it succeeds on. I think these get built when the engine is built but i'm not certain. I've been trying to figure out a good way to keep the container around once this fails to further inspect things. Is that possible?
I was able to spin up a container from the one that failed and I made a little progress, confirmed it is a missing DLL. Also this seems to be intermittent since the build is failing on my desktop too now.
C:\UnrealEngine>dlldiag deps "C:\UnrealEngine\Engine\Binaries\Win64\UnrealPak.exe"
DLL Diagnostic Tools version 0.0.18
Copyright (c) 2019-2021 Adam Rehn
Parsing module header and identifying direct dependencies... done.
Parsed module details:
Module: C:\UnrealEngine\Engine\Binaries\Win64\UnrealPak.exe
Type: Executable
Architecture: x64
Attempting to load the module's direct dependencies:
api-ms-win-crt-heap-l1-1-0.dll Loaded successfully
api-ms-win-crt-locale-l1-1-0.dll Loaded successfully
api-ms-win-crt-math-l1-1-0.dll Loaded successfully
api-ms-win-crt-runtime-l1-1-0.dll Loaded successfully
api-ms-win-crt-stdio-l1-1-0.dll Loaded successfully
api-ms-win-crt-string-l1-1-0.dll Loaded successfully
api-ms-win-crt-utility-l1-1-0.dll Loaded successfully
dbghelp.dll Loaded successfully
KERNEL32.dll Loaded successfully
UnrealPak-ApplicationCore.dll Error 126: The specified module could not be found.
UnrealPak-Core.dll Loaded successfully
[547/2787] Compile [x64] Module.ApplicationCore.cpp
[548/2787] Link [x64] UnrealInsights-TraceAnalysis.lib
Creating library ..\Intermediate\Build\Win64\x64\UnrealInsights\Development\TraceAnalysis\UnrealInsights-TraceAnalysis.lib and object ..\Intermediate\Build\Win64\x64\UnrealInsights\Development\TraceAnalysis\UnrealInsights-TraceAnalysis.exp
[549/2787] Link [x64] UnrealPak-ApplicationCore.lib
Creating library ..\Intermediate\Build\Win64\x64\UnrealPak\Development\ApplicationCore\UnrealPak-ApplicationCore.lib and object ..\Intermediate\Build\Win64\x64\UnrealPak\Development\ApplicationCore\UnrealPak-ApplicationCore.exp
[550/2787] Compile [x64] Module.Slate.1.cpp
[551/2787] Compile [x64] Module.Sockets.cpp
[552/2787] Compile [x64] Module.Json.cpp
[553/2787] Compile [x64] Module.Cbor.cpp
[554/2787] Link [x64] UnrealPak-ApplicationCore.dll
Creating library ..\Intermediate\Build\Win64\x64\UnrealPak\Development\ApplicationCore\UnrealPak-ApplicationCore.sup.lib and object ..\Intermediate\Build\Win64\x64\UnrealPak\Development\ApplicationCore\UnrealPak-ApplicationCore.sup.exp
C:\UnrealEngine>dir Engine\Intermediate\Build\Win64\x64\UnrealPak\Development\ApplicationCore
Volume in drive C has no label.
Volume Serial Number is DAFF-C33C
Directory of C:\UnrealEngine\Engine\Intermediate\Build\Win64\x64\UnrealPak\Development\ApplicationCore
01/07/2024 10:13 AM <DIR> .
01/07/2024 10:15 AM <DIR> ..
01/07/2024 10:11 AM 823 ApplicationCore.Shared.rsp
01/07/2024 10:13 AM 24,912 Default.rc2.res
01/07/2024 10:11 AM 2,304 Default.rc2.res.rsp
01/07/2024 10:11 AM 714 Definitions.ApplicationCore.h
01/07/2024 10:11 AM 2,829 Module.ApplicationCore.cpp
01/07/2024 10:13 AM 42,292 Module.ApplicationCore.cpp.dep.json
01/07/2024 10:13 AM 5,405,939 Module.ApplicationCore.cpp.obj
01/07/2024 10:11 AM 1,337 Module.ApplicationCore.cpp.obj.rsp
01/07/2024 10:11 AM 1,967 UnrealPak-ApplicationCore.dll.rsp
01/07/2024 10:13 AM 97,838 UnrealPak-ApplicationCore.exp
01/07/2024 10:13 AM 164,240 UnrealPak-ApplicationCore.lib
01/07/2024 10:11 AM 526 UnrealPak-ApplicationCore.lib.rsp
01/07/2024 10:13 AM 97,842 UnrealPak-ApplicationCore.sup.exp
01/07/2024 10:13 AM 164,240 UnrealPak-ApplicationCore.sup.lib
14 File(s) 6,007,803 bytes
2 Dir(s) 965,989,371,904 bytes free
Also maybe relevant? Although these errors also appear in logs for when the builds worked. This is during the prerequisites build:
Installing windows-sdk-10-version-1809-windbg...
windows-sdk-10-version-1809-windbg has been installed.
windows-sdk-10-version-1809-windbg may be able to be automatically uninstalled.
The install of windows-sdk-10-version-1809-windbg was successful.
Software installed as 'EXE', install location is likely default.
Chocolatey installed 17/20 packages. 3 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Installed:
- vcredist2013 v12.0.40660.20180427
- vcredist140 v14.38.33130
- python v3.12.1
- vcredist-all v1.0.1
- kb3033929 v1.0.5
- vcredist2012 v11.0.61031.20230518
- kb3035131 v1.0.3
- kb2919355 v1.0.20160915
- python312 v3.12.1
- kb2919442 v1.0.20160915
- kb2999226 v1.0.20181019
- windows-sdk-10-version-1809-windbg v10.0.17763.132
- vcredist2017 v14.16.27033
- choco-cleaner v1.1.1
- vcredist2015 v14.0.24215.20170201
- chocolatey-windowsupdate.extension v1.0.5
- python3 v3.12.1
Failures
- vcredist2008 (exited 1) - Error while running 'C:\ProgramData\chocolatey\lib\vcredist2008\tools\chocolateyInstall.ps1'.
See log for details.
- vcredist2010 (exited 1) - Error while running 'C:\ProgramData\chocolatey\lib\vcredist2010\tools\chocolateyInstall.ps1'.
See log for details.
- vcredist2005 (exited -2147023829) - Error while running 'C:\ProgramData\chocolatey\lib\vcredist2005\tools\chocolateyInstall.ps1'.
See log for details.
manager
Microsoft (R) File Expansion Utility
Copyright (c) Microsoft Corporation. All rights reserved.
Can't open input file: c:\users\containeradministrator\appdata\local\temp\apr2007_xinput_x64.cab.
Microsoft (R) File Expansion Utility
Copyright (c) Microsoft Corporation. All rights reserved.
Can't open input file: c:\users\containeradministrator\appdata\local\temp\jun2010_d3dcompiler_43_x64.cab.
Microsoft (R) File Expansion Utility
Copyright (c) Microsoft Corporation. All rights reserved.
Can't open input file: c:\users\containeradministrator\appdata\local\temp\feb2010_x3daudio_x64.cab.
Microsoft (R) File Expansion Utility
Copyright (c) Microsoft Corporation. All rights reserved.
Can't open input file: c:\users\containeradministrator\appdata\local\temp\jun2010_xaudio_x64.cab.
Microsoft (R) File Expansion Utility
Copyright (c) Microsoft Corporation. All rights reserved.
Can't open input file: c:\users\containeradministrator\appdata\local\temp\jun2010_xaudio_x64.cab.
Directory: C:\Users\ContainerAdministrator\AppData\Local
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 1/7/2024 9:15 AM Temp
So, are you trying to say that errors do not abort ue4-build-prerequisites image?
@slonopotamus appears that way, at least for these errors. But like I said I've successfully built the minimal image despite seeing those errors in ue4-build-prerequisites
More progress, though I don't entirely know why its failing to find these dlls. dlldiag trace command errors out when I try to trace UnrealPak-ApplicationCore.dll
PS C:\UnrealEngine> dlldiag deps .\Engine\Binaries\Win64\UnrealPak-ApplicationCore.dll
DLL Diagnostic Tools version 0.0.18
Copyright (c) 2019-2021 Adam Rehn
Parsing module header and identifying direct dependencies... done.
Parsed module details:
Module: C:\UnrealEngine\Engine\Binaries\Win64\UnrealPak-ApplicationCore.dll
Type: Dynamic-Link Library
Architecture: x64
Attempting to load the module's direct dependencies:
ADVAPI32.dll Loaded successfully
api-ms-win-crt-convert-l1-1-0.dll Loaded successfully
api-ms-win-crt-heap-l1-1-0.dll Loaded successfully
api-ms-win-crt-math-l1-1-0.dll Loaded successfully
api-ms-win-crt-runtime-l1-1-0.dll Loaded successfully
api-ms-win-crt-stdio-l1-1-0.dll Loaded successfully
api-ms-win-crt-string-l1-1-0.dll Loaded successfully
dwmapi.dll Loaded successfully
dxgi.dll Loaded successfully
GDI32.dll Loaded successfully
IMM32.dll Loaded successfully
KERNEL32.dll Loaded successfully
ole32.dll Loaded successfully
OLEAUT32.dll Loaded successfully
POWRPROF.dll Loaded successfully
SETUPAPI.dll Loaded successfully
UnrealPak-Core.dll Loaded successfully
USER32.dll Loaded successfully
UxTheme.dll Loaded successfully
VCRUNTIME140.dll Loaded successfully
VCRUNTIME140_1.dll Loaded successfully
XINPUT1_3.dll Error 126: The specified module could not be found.
PS C:\> dlldiag trace C:\UnrealEngine\Engine\Binaries\ThirdParty\AppLocalDependencies\Win64\DirectX\xinput1_3.dll
DLL Diagnostic Tools version 0.0.18
Copyright (c) 2019-2021 Adam Rehn
Parsing module header and identifying non delay-loaded dependencies... done.
Identifying the module's delay-loaded dependencies... done.
Parsed module details:
Module: C:\UnrealEngine\Engine\Binaries\ThirdParty\AppLocalDependencies\Win64\DirectX\xinput1_3.dll
Type: Dynamic-Link Library
Architecture: x64
The module imports 4 direct dependencies:
ADVAPI32.dll
KERNEL32.dll
ntdll.dll
SETUPAPI.dll
Performing LoadLibrary() trace for C:\UnrealEngine\Engine\Binaries\ThirdParty\AppLocalDependencies\Win64\DirectX\xinput1_3.dll...
Performing LoadLibrary() trace for ADVAPI32.dll...
Performing LoadLibrary() trace for KERNEL32.dll...
Performing LoadLibrary() trace for ntdll.dll...
Performing LoadLibrary() trace for SETUPAPI.dll...
Done.
Summary of LdrLoadDll calls:
ADVAPI32.dll Loaded successfully
C:\UnrealEngine\Engine\Binaries\ThirdParty\AppLocalDependencies\Win64\DirectX\xinput1_3.dll Loaded successfully
KERNEL32.dll Loaded successfully
ntdll.dll Loaded successfully
SETUPAPI.dll Loaded successfully
Summary of LdrpLoadDllInternal calls:
ADVAPI32.dll Loaded successfully
C:\UnrealEngine\Engine\Binaries\ThirdParty\AppLocalDependencies\Win64\DirectX\xinput1_3.dll Loaded successfully
KERNEL32.dll Loaded successfully
ntdll.dll Loaded successfully
SETUPAPI.dll Loaded successfully
Summary of LdrpMinimalMapModule calls:
C:\UnrealEngine\Engine\Binaries\ThirdParty\AppLocalDependencies\Win64\DirectX\xinput1_3.dll Mapped successfully
C:\Windows\System32\ADVAPI32.dll Mapped successfully
C:\Windows\System32\msvcrt.dll Mapped successfully
C:\Windows\System32\RPCRT4.dll Mapped successfully
C:\Windows\System32\sechost.dll Mapped successfully
C:\Windows\System32\SETUPAPI.dll Mapped successfully
Summary of LdrpResolveDllName calls:
xinput1_3.dll C:\UnrealEngine\Engine\Binaries\ThirdParty\AppLocalDependencies\Win64\DirectX\xinput1_3.dll
PS C:\> dlldiag trace C:\UnrealEngine\Engine\Binaries\Win64\UnrealPak-ApplicationCore.dll
DLL Diagnostic Tools version 0.0.18
Copyright (c) 2019-2021 Adam Rehn
Parsing module header and identifying non delay-loaded dependencies... done.
Identifying the module's delay-loaded dependencies... done.
Parsed module details:
Module: C:\UnrealEngine\Engine\Binaries\Win64\UnrealPak-ApplicationCore.dll
Type: Dynamic-Link Library
Architecture: x64
The module imports 22 direct dependencies:
ADVAPI32.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
dwmapi.dll
dxgi.dll
GDI32.dll
IMM32.dll
KERNEL32.dll
ole32.dll
OLEAUT32.dll
POWRPROF.dll
SETUPAPI.dll
UnrealPak-Core.dll
USER32.dll
UxTheme.dll
VCRUNTIME140.dll
VCRUNTIME140_1.dll
XINPUT1_3.dll
Performing LoadLibrary() trace for C:\UnrealEngine\Engine\Binaries\Win64\UnrealPak-ApplicationCore.dll...
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Python312\Scripts\dlldiag.exe\__main__.py", line 7, in <module>
File "C:\Python312\Lib\site-packages\dlldiag\main.py", line 29, in main
subcommands[subcommand]['function']()
File "C:\Python312\Lib\site-packages\dlldiag\subcommands\trace.py", line 226, in trace
result = TraceHelpers.performTrace(debugger, helper, module, architecture, cwd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python312\Lib\site-packages\dlldiag\subcommands\trace.py", line 141, in performTrace
match.result = ntdll.RtlNtStatusToDosError(int(parsed['result'], 16))
^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: int() can't convert non-string with explicit base
I would assume C:\UnrealEngine\Engine\Binaries\ThirdParty\AppLocalDependencies\Win64\DirectX isn't in the path, so executables can't see it. I expect something is supposed to install stuff from there, but I don't have the code at hand so can't say for sure.
The failure with apr2007_xinput_x64.cab
definitely looks suspicious to me, since that code is extracting xinput1_3.dll. I'm not sure how things worked when this failed... Maybe sometimes some other stage of UE build is doing a DirectX Redistributable install for you.
In the working image, does _C:\Windows\System32\xinput13.dll exist? What does dlldiag trace C:\UnrealEngine\Engine\Binaries\Win64\UnrealPak-ApplicationCore.dll
report? (Assuming it doesn't crash there too.)
Anyway, I can see two apparent bugs here, but neither one seems causal.
expand
failed but didn't abort the build.dlldiag trace
call. (I suspect a None was passed in there but can't see how that happened.)In the working image _C:\Windows\System32\xinput13.dll does exist. Unfortunately the trace also fails on the working container.
Is there anything else I could try to get to help debug this further? Does it sound like the issue is maybe with the prerequisites container? Unfortunately I don't have a log of the prereq container being built prior to a successful build. I could try building that a few more times to see if those expand errors are intermittent. I could also try to rebuild UnrealPak.exe to see if that is failing? I'm not sure the command syntax to do that.
I appreciate the help!
Just to be clear, _C:\Windows\System32\xinput13.dll doesn't exist in the failed image, do I have that correct? The dlldiag trace
is probably an unrelated problem, possibly to do with how UnreakPak-ApplicationCore.dll
works in some way. It probably tries to import the host executable or something.
So yeah, I'm not sure, but it seems like rebuilding ue-prerequisites container and checking the log to ensure the files are extracted would be the go. We definitely need to fix the build passing with those errors, we probably need an errorlevel ($?
or $LASTEXITCODE
) check after each call to expand
, or maybe even explicitly checking that the target file exists.
Even better will be working out why those lines are failing. We probably also need to check the result of running the redist, as I would guess that the problem lies there somehow, and it failed to extract the cab files we need.
I just tried it locally (outside a container) and it seemed to work fine, so it's probably not a problem with the redist itself. But it did pop up a UI during extraction, which hopefully isn't causing issues. (I don't think that's changed either.)
Maybe I am misunderstanding PowerShell error handling... I thought that this would be enough to abort everything in case of an error.
It looks like PowerShell 7.3 introduced $PSNativeCommandUseErrorActionPreference
which controls whether native executables returning non-zero errorlevels are handled by $ErrorActionPreference
and it's on by default; but I'm not sure what the behaviour in PowerShell 5.1 was. The docs suggest it's a new behaviour and PowerShell 5.1 didn't try to handle them.
So if you tested the script on PowerShell 7.3 or later, it'd probably have stopped as desired.
@TBBle _C:\Windows\System32\xinput13.dll does not exist on the failed image. I gather that after a successful prerequisites build, it should. I'll do a few runs today and see if I can grab the log of when it gets placed in that location.
edited for correctness More progress and I at least found a reliable workaround. Running the build with --isolation=hyperv
allows those three vcredist packages to install correctly! This is true on both my Windows 11 Pro desktop and Windows 11 Pro VM. And C:\Windows\System32\xinput1_3.dll
is present. This isn't too big of an issue for me since I probably wont need to rebuild the prereq image often and I can just target that image and build the rest with process isolation.
When installing those problematic packages with --isolation=process
the installers error out immediately with exit code of 1 but no other information.
I originally didn't specify the --isolation
parameter and it was defaulting to hyperv for me. I then saw how slow compilation was going on the source image and figured out i should use process isolation. Since I had already built the build-prerequisites image with hyperv isolation, everything worked. When i went and built all the images from scratch on my vm, I just copied the command i used which included the --isolation=process
parameter and so I wasn't able to build the build-prerequisites image correctly.
When installing those problematic packages with
--isolation=hyperv
the installers error out immediately with exit code of 1 but no other information.
Did you mean --isolation=process
here?
Anyway, it's fantastic that we've identified a workaround. My suspicion is that this is something that only affects Windows 11, as I recall we've seen weird issues with Windows 11 Process Isolation running Server 2022 images, and ended up floating those over to Microsoft to investigate.
Are you by any chance using an AMD CPU?
Thanks - updated my last comment to correct the typos. My CPU is an AMD 7970x Threadripper. Do you think that has something to do with it? Perhaps OP has something similar. Their ue4-docker info shows 24 physical cores.
Yup, the original poster shows AuthenticAMD
on Windows 11 (22621), so the underlying failure may well be another symptom of those. I really must remember to check for that first when looking for failures... The MS side-bug is closed, but I don't see any official MS commentary describing the fix.
So.
Thanks for the help with this @TBBle and @slonopotamus !!
Output of the
ue4-docker info
command:Additional details:
We are trying to build a Windows docker container with the full Unreal Engine for CI/CD but for the life of me can't get passed the "Make Feature Packs" and it seems that not even an error logfile gets produces based from it saying "None!, no output produced". I have tried both vs2019/vs2022 as well as both the built in 5.1 build and the custom build with the url to the UE 5.1 branch. Below is the command I use as well as my system info (Actually disk space available is 1.1 TB, but we are copying some files over at the time of capture). Any help is appreciated!
Command: