`choco install` fails with `Exit code was '-2147023829'` on a Windows 11 host with process isolation

Output of the ue4-docker info command:

PS D:\ue4docker> ue4-docker info ue4-docker version: 0.0.94 (latest available version is 0.0.94) Operating system: Windows 10 Pro (Build 22000.376) Docker daemon version: 20.10.11 NVIDIA Docker supported: No Maximum image size: 400GB Available disk space: 6.7 TiB Total system memory: 255.83 GiB physical, 293.83 GiB virtual Number of processors: 64 physical, 128 logical

Additional details:

Are you accessing the network through a proxy server? NO

I am running windows 11 as a host OS.

I am building using the following command: PS D:\ue4docker> ue4-docker build -username redacted -password redacted --exclude ddc -basetag ltsc2022 -isolation=process --visual-studio=2019 4.27.2

The build is failing to create the prerequisites image, when installing git from chocolatey. ERROR: Running ["C:\ProgramData\chocolatey\lib\git.install\tools\Git-2.34.1-64-bit.exe" /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /NOCANCEL /SP- /LOG /COMPONENTS="icons,assoc,assoc_sh,,,,gitlfs,icons\quicklaunch" /o:PathOption=Cmd /o:BashTerminalOption=ConHost /o:CRLFOption=CRLFCommitAsIs ] was not successful. Exit code was '-1073741819'. See log for possible error messages.

I tried creating a ltsc2022 windows server container, and repeated the following command: choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"

git installed successfully during that attempt, so this does not appear to be a chocolatey issue.

I have repeated the ue4-docker build several times, and get the same result. ue4-docker build log.txt chocolatey.log

That's an access violation (0xc0000005), which is a super-weird thing to happen, and certainly shouldn't be caused by any of the things done to create the container...

If you run the container image as it stood before that step (d3f3e304f75d in your attached build log), and run the same choco install command again, does it fail?

If so, then something done earlier in the image build has corrupted the system weirdly, but since all that has been done is copy some DLLs, enable long paths, and install chocolately, I can't see what it would be.

If not, then something in how ue4-docker is starting the container must be different from when it's run by hand.

Was your stand-alone test also using process isolation, by the way? It's not the default for docker run on client Windows builds.

Hi, thanks for looking into this, you are right, I must have not specified isolation during my initial test. I have now repeated some testing, and it appears that the issue occurs on both server core and prerequisites containers when running in process isolation, but not with the default isolation behaviour.

Test1: Existing prerequisites container

I create a container from the image as it stood before the last step using the following command: docker create --isolation=process --interactive d3f3e304f75d I manually execute the command:

'choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"'

The result is:
`C:\>choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"
choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"
Chocolatey v0.11.3
Installing the following packages:
git
By installing, you accept licenses for the packages.
Progress: Downloading git.install 2.34.1... 100%
Progress: Downloading chocolatey-core.extension 1.3.5.1... 100%
Progress: Downloading git 2.34.1... 100%

chocolatey-core.extension v1.3.5.1 [Approved]
chocolatey-core.extension package files install completed. Performing other installation steps.
 Installed/updated chocolatey-core extensions.
 The install of chocolatey-core.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'

git.install v2.34.1 [Approved]
git.install package files install completed. Performing other installation steps.
Parameter: no git shell integration
Using Git LFS
Installing 64-bit git.install...
ERROR: Running ["C:\ProgramData\chocolatey\lib\git.install\tools\Git-2.34.1-64-bit.exe" /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /NOCANCEL /SP- /LOG /COMPONENTS="icons,assoc,assoc_sh,,,,gitlfs,icons\quicklaunch" /o:PathOption=Cmd /o:BashTerminalOption=ConHost /o:CRLFOption=CRLFCommitAsIs ] was not successful. Exit code was '-1073741819'. See log for possible error messages.
The install of git.install was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\git.install\tools\chocolateyInstall.ps1'.
 See log for details.

git v2.34.1 [Approved]
git package files install completed. Performing other installation steps.
 The install of git was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

Chocolatey installed 2/3 packages. 1 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Failures
 - git.install (exited -1073741819) - Error while running 'C:\ProgramData\chocolatey\lib\git.install\tools\chocolateyInstall.ps1'.
 See log for details.`
### Test 2: server core container with process isolation
Next I repeat the process with server core:
'mcr.microsoft.com/windows/servercore   ltsc2022   136110e1a05c   4 weeks ago    4.91GB'

'docker create --isolation=process --interactive 136110e1a05c
docker start bf9e8e81f63baa5d67d10d4d01c3105e3f3952f16f04be2cf05d5aa35fed72ee
docker attach bf9e8e81f63baa5d67d10d4d01c3105e3f3952f16f04be2cf05d5aa35fed72ee'

I install chocolatey using powerscript, and run the command again:
'C:\>powershell
powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\> Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
Forcing web requests to allow TLS v1.2 (Required for requests to Chocolatey.org)
Getting latest version of the Chocolatey package for download.
Not using proxy.
Getting Chocolatey from https://community.chocolatey.org/api/v2/package/chocolatey/0.11.3.
Downloading https://community.chocolatey.org/api/v2/package/chocolatey/0.11.3 to C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip
Not using proxy.
Extracting C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip to C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall
Installing Chocolatey on the local machine
Creating ChocolateyInstall as an environment variable (targeting 'Machine')
  Setting ChocolateyInstall to 'C:\ProgramData\chocolatey'
WARNING: It's very likely you will need to close and reopen your shell
  before you can use choco.
Restricting write permissions to Administrators
We are setting up the Chocolatey package repository.
The packages themselves go to 'C:\ProgramData\chocolatey\lib'
  (i.e. C:\ProgramData\chocolatey\lib\yourPackageName).
A shim file for the command line goes to 'C:\ProgramData\chocolatey\bin'
  and points to an executable in 'C:\ProgramData\chocolatey\lib\yourPackageName'.

Creating Chocolatey folders if they do not already exist.

WARNING: You can safely ignore errors related to missing log files when
  upgrading from a version of Chocolatey less than 0.9.9.
  'Batch file could not be found' is also safe to ignore.
  'The system cannot find the file specified' - also safe.
chocolatey.nupkg file not installed in lib.
 Attempting to locate it from bootstrapper.
PATH environment variable does not have C:\ProgramData\chocolatey\bin in it. Adding...
WARNING: Not setting tab completion: Profile file does not exist at
'C:\Users\ContainerAdministrator\Documents\WindowsPowerShell\Microsoft.PowerShe
ll_profile.ps1'.
Chocolatey (choco.exe) is now ready.
You can call choco from anywhere, command line or powershell by typing choco.
Run choco /? for a list of functions.
You may need to shut down and restart powershell and/or consoles
 first prior to using choco.
Ensuring Chocolatey commands are on the path
Ensuring chocolatey.nupkg is in the lib folder
PS C:\> choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"
choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"
Chocolatey v0.11.3
Installing the following packages:
git
By installing, you accept licenses for the packages.
Progress: Downloading git.install 2.34.1... 100%
Progress: Downloading chocolatey-core.extension 1.3.5.1... 100%
Progress: Downloading git 2.34.1... 100%

chocolatey-core.extension v1.3.5.1 [Approved]
chocolatey-core.extension package files install completed. Performing other installation steps.
 Installed/updated chocolatey-core extensions.
 The install of chocolatey-core.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'

git.install v2.34.1 [Approved]
git.install package files install completed. Performing other installation steps.
Parameter: no git shell integration
Using Git LFS
Installing 64-bit git.install...
ERROR: Running ["C:\ProgramData\chocolatey\lib\git.install\tools\Git-2.34.1-64-bit.exe" /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /NOCANCEL /SP- /LOG /COMPONENTS="icons,assoc,assoc_sh,,,,gitlfs,icons\quicklaunch" /o:PathOption=Cmd /o:BashTerminalOption=ConHost /o:CRLFOption=CRLFCommitAsIs ] was not successful. Exit code was '-1073741819'. See log for possible error messages.
The install of git.install was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\git.install\tools\chocolateyInstall.ps1'.
 See log for details.

git v2.34.1 [Approved]
git package files install completed. Performing other installation steps.
 The install of git was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

Chocolatey installed 2/3 packages. 1 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Failures
 - git.install (exited -1073741819) - Error while running 'C:\ProgramData\chocolatey\lib\git.install\tools\chocolateyInstall.ps1'.
 See log for details.
PS C:\>'

Test 3: server core with default isolation

Creating the same image, but without specifying process isolation:

'docker create --interactive 136110e1a05c
docker start 52442b3743bd5b24829e3b43520b5e67954dc28babf03b95087c9a2aa8609b9c
docker attach 52442b3743bd5b24829e3b43520b5e67954dc28babf03b95087c9a2aa8609b9c'

'C:\>powershell
powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\> Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
Forcing web requests to allow TLS v1.2 (Required for requests to Chocolatey.org)
Getting latest version of the Chocolatey package for download.
Not using proxy.
Getting Chocolatey from https://community.chocolatey.org/api/v2/package/chocolatey/0.11.3.
Downloading https://community.chocolatey.org/api/v2/package/chocolatey/0.11.3 to C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip
Not using proxy.
Extracting C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip to C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall
Installing Chocolatey on the local machine
Creating ChocolateyInstall as an environment variable (targeting 'Machine')
  Setting ChocolateyInstall to 'C:\ProgramData\chocolatey'
WARNING: It's very likely you will need to close and reopen your shell
  before you can use choco.
Restricting write permissions to Administrators
We are setting up the Chocolatey package repository.
The packages themselves go to 'C:\ProgramData\chocolatey\lib'
  (i.e. C:\ProgramData\chocolatey\lib\yourPackageName).
A shim file for the command line goes to 'C:\ProgramData\chocolatey\bin'
  and points to an executable in 'C:\ProgramData\chocolatey\lib\yourPackageName'.

Creating Chocolatey folders if they do not already exist.

WARNING: You can safely ignore errors related to missing log files when
  upgrading from a version of Chocolatey less than 0.9.9.
  'Batch file could not be found' is also safe to ignore.
  'The system cannot find the file specified' - also safe.
chocolatey.nupkg file not installed in lib.
 Attempting to locate it from bootstrapper.
PATH environment variable does not have C:\ProgramData\chocolatey\bin in it. Adding...
WARNING: Not setting tab completion: Profile file does not exist at
'C:\Users\ContainerAdministrator\Documents\WindowsPowerShell\Microsoft.PowerShe
ll_profile.ps1'.
Chocolatey (choco.exe) is now ready.
You can call choco from anywhere, command line or powershell by typing choco.
Run choco /? for a list of functions.
You may need to shut down and restart powershell and/or consoles
 first prior to using choco.
Ensuring Chocolatey commands are on the path
Ensuring chocolatey.nupkg is in the lib folder
PS C:\> choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"
choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"
Chocolatey v0.11.3
Installing the following packages:
git
By installing, you accept licenses for the packages.
Progress: Downloading git.install 2.34.1... 100%
Progress: Downloading chocolatey-core.extension 1.3.5.1... 100%
Progress: Downloading git 2.34.1... 100%

chocolatey-core.extension v1.3.5.1 [Approved]
chocolatey-core.extension package files install completed. Performing other installation steps.
 Installed/updated chocolatey-core extensions.
 The install of chocolatey-core.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'

git.install v2.34.1 [Approved]
git.install package files install completed. Performing other installation steps.
Parameter: no git shell integration
Using Git LFS
Installing 64-bit git.install...
git.install has been installed.
WARNING: Can't find git.install install location
  git.install can be automatically uninstalled.
Environment Vars (like PATH) have changed. Close/reopen your shell to
 see the changes (or in powershell/cmd.exe just type `refreshenv`).
 The install of git.install was successful.
  Software installed to 'C:\Program Files\Git\'

git v2.34.1 [Approved]
git package files install completed. Performing other installation steps.
 The install of git was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

Chocolatey installed 3/3 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
PS C:\>'

Test4: reinstalling git on process isolation windows core container

Connecting back to the process isolation container with windows core, I then started to investigate how the installer fails. I noticed that despite the failure, the container has a working copy of git installed.

I ran the following command to pull a fresh copy of the installer:

'Invoke-WebRequest -Uri https://github.com/git-for-windows/git/releases/download/v2.34.1.windows.1/Git-2.34.1-64-bit.exe -OutFile ./Git-2.34.1-64-bit.exe'

Then i manually uninstalled git:
"%ProgramFiles%\Git\unins000.exe" /VERYSILENT /NORESTART

then running the installer with a log file specified:
'C:\>Git-2.34.1-64-bit.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /NOCANCEL /SP- /LOG="c:\test2.log" /COMPONENTS="icons,assoc,assoc_sh,,,,gitlfs,icons\quicklaunch" /o:PathOption=Cmd /o:BashTerminalOption=ConHost /o:CRLFOption=CRLFCommitAsIs'

Log appears to show a successful install, i will repeat this process on a fresh image and see if I can get it to fall over. test2.log

Ok, manually downloading and installing git on a fresh windows core container with process isolation appears to work fine. The install completes without any errors.

Looking at the containers where installs fail, they still end up with working copies of Git installed, so something must be causing the installer to fail late in the install process. As this is happening both on server core and prerequisites containers when installing with chocolatey, but succeeds when installed manually on server core, I would assume the cause of the issue is likely to do with the chocolatey installer package.

Test5: install git on new server core image with process isolation

D:\ue4docker>docker create --isolation=process --interactive 136110e1a05c
fd75a9adc012bcfbd397ec52a656a6f31250892ddf4ef0b1d73ff766021434c5

D:\ue4docker>docker start fd75a9adc012bcfbd397ec52a656a6f31250892ddf4ef0b1d73ff766021434c5
fd75a9adc012bcfbd397ec52a656a6f31250892ddf4ef0b1d73ff766021434c5

D:\ue4docker>docker attach fd75a9adc012bcfbd397ec52a656a6f31250892ddf4ef0b1d73ff766021434c5
powershell
powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\> Invoke-WebRequest -Uri https://github.com/git-for-windows/git/releases/download/v2.34.1.windows.1/Git-2.34.1-64-bit.exe -OutFile ./Git-2.34.1-64-bit.exe
Invoke-WebRequest -Uri https://github.com/git-for-windows/git/releases/download/v2.34.1.windows.1/Git-2.34.1-64-bit.exe -OutFile ./Git-2.34.1-64-bit.exe
PS C:\> exit
exit

C:\>Git-2.34.1-64-bit.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /NOCANCEL /SP- /LOG="c:\test.log" /COMPONENTS="icons,assoc,assoc_sh,,,,gitlfs,icons\quicklaunch" /o:PathOption=Cmd /o:BashTerminalOption=ConHost /o:CRLFOption=CRLFCommitAsIs
Git-2.34.1-64-bit.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /NOCANCEL /SP- /LOG="c:\test.log" /COMPONENTS="icons,assoc,assoc_sh,,,,gitlfs,icons\quicklaunch" /o:PathOption=Cmd /o:BashTerminalOption=ConHost /o:CRLFOption=CRLFCommitAsIs

Log file: test.log

I'm suspicious of that "success" even in the Hyper-V isolation mode, due to

WARNING: Can't find git.install install location

which comes from the second-last line of this snippet from the install script, where the first line in the snippet is the failing line in process isolation:

Install-ChocolateyInstallPackage @packageArgs
Get-ChildItem $toolsPath\$fileName32, $toolsPath\$fileName64 | ForEach-Object { Remove-Item $_ -ea 0; if (Test-Path $_) { Set-Content "$_.ignore" '' } }

$packageName = $packageArgs.packageName
$installLocation = Get-AppInstallLocation $packageArgs.SoftwareName
if (!$installLocation)  { Write-Warning "Can't find $packageName install location"; return }
Write-Host "$packageName installed to '$installLocation'"

I also noted on the Chocolatey package discussion, a few people have had issues over the years when the installer fails if doing a silent install and it wants to close some running programs. However, I super-doubt that's the case here (different error code, and what would be running??)

Anyway, since this repros on Windows Server Core without ue4-docker, then I'd suggest raising it to Chocolatey (not sure what their support channels are), and/or perhaps the Windows Containers issue tracker since this seems to be affected by the isolation mode, suggesting it's a container issue at its core.

To split the difference in the possible problems, try repeating Test 5, but run

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

before downloading and running the git installer manually in the same container instance. That will help identify if it's something about Chocolatey itself, or something about the git package specifically.

Another possible test you might want to try in the meantime, is using a DockerFile that starts from server core, and installs chocolatey and then the git package as two different RUN steps, rather than one RUN step like ue4-docker does, to emulate a system restart and clear any file-in-use issues.

Thanks for the help.

I have opened a ticket with chocolatey relating to this issue here: https://github.com/chocolatey-community/chocolatey-packages/issues/1776

On that issue I have built a dockerfile that installs chocolatey, then choco git as a separate RUN line. This errors out with process isolation, and installs otherwise.

Weirdly the git installer seems to produce some error text when run in a RUN statement from a dockerfile. It complains about the extra commas in /Components, and return 1. I see none of this when running the commands interactively. The error text is displayed on either isolation mode. Dockerfile.txt

Hmm... I wonder which part (Windows, Chocolatey, Git, Docker?) has changed. I successfully built ltsc2022 back in September when was working on #199. One major difference I see is that I was using Windows Server 2022 as a host instead of Windows 11, but this might be irrelevant. Maybe you could to try older Git versions?

I can probably rule out Git, I have tried every version since 2.31.1, and get the same result on each. Windows host sounded like an interesting avenue, so I spun up a VM and see if I get the same problems with server 22 as host OS. The same docker file that fails with process isolation with win11 as host, succeeds with server 2022 as host.

This was done with the latest choco, and git releases. Interestingly the same git.install install location error is showing, but git does get installed correctly.

docker build output

PS C:\Users\testdocker\gittest> docker build --isolation=process .
Sending build context to Docker daemon  3.072kB
Step 1/5 : FROM mcr.microsoft.com/windows/servercore:ltsc2022
ltsc2022: Pulling from windows/servercore
8f616e6e9eec: Pull complete
4d1d74adc6a9: Pull complete
Digest: sha256:1f8965b9b4e2097ac78590ba652f069eafd5b6a0a4e9e603a415d362bfb7638c
Status: Downloaded newer image for mcr.microsoft.com/windows/servercore:ltsc2022
 ---> 136110e1a05c
Step 2/5 : SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'Continue'; $verbosePreference='Continue';"]
 ---> Running in 3d4652165df7
Removing intermediate container 3d4652165df7
 ---> dcc0da35f0c0
Step 3/5 : ENV CHOCO_URL=https://chocolatey.org/install.ps1
 ---> Running in 4e407a001325
Removing intermediate container 4e407a001325
 ---> fe48d8a5cf2a
Step 4/5 : RUN Set-ExecutionPolicy Bypass -Scope Process -Force;  [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12';  iex ((New-Object System.Net.WebClient).DownloadString("$env:CHOCO_URL"));
 ---> Running in 774f8ed5f97b
Forcing web requests to allow TLS v1.2 (Required for requests to Chocolatey.org)
Getting latest version of the Chocolatey package for download.
Not using proxy.
Getting Chocolatey from https://community.chocolatey.org/api/v2/package/chocolatey/0.11.3.
Downloading https://community.chocolatey.org/api/v2/package/chocolatey/0.11.3 to C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip
Not using proxy.
Extracting C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip to C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall
VERBOSE: Preparing to expand...
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\_re
ls\.rels'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\cho
colatey.nuspec'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateysetup.psm1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\init.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\VERIFICATION.txt'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\choco.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\choco.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\choco.exe.manifest'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\CREDITS.txt'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\LICENSE.txt'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\chocolateyInstaller.psm1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\chocolateyProfile.psm1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\chocolateyScriptRunner.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\ChocolateyTabExpansion.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Format-FileSize.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-CheckSumValid.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-ChocolateyUnzip.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-ChocolateyWebFile.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-EnvironmentVariable.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-EnvironmentVariableNames.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-FtpFile.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-OSArchitectureWidth.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-PackageParameters.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-ToolsLocation.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-UACEnabled.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-UninstallRegistryKey.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-VirusCheckValid.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-WebFile.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-WebFileName.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Get-WebHeaders.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-BinFile.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyDesktopLink.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyEnvironmentVariable.ps
1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyExplorerMenuItem.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyFileAssociation.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyInstallPackage.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyPackage.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyPath.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyPinnedTaskBarItem.ps1'
.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyPowershellCommand.ps1'
.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyShortcut.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyVsixPackage.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-ChocolateyZipPackage.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Install-Vsix.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Set-EnvironmentVariable.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Set-PowerShellExitCode.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Start-ChocolateyProcessAsAdmin.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Test-ProcessAdminRights.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Uninstall-BinFile.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Uninstall-ChocolateyEnvironmentVariable.
ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Uninstall-ChocolateyPackage.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\UnInstall-ChocolateyZipPackage.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Update-SessionEnvironment.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Write-ChocolateyFailure.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Write-ChocolateySuccess.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Write-FileUpdateLog.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\functions\Write-FunctionCallLogMessage.ps1'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\choco.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\choco.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\chocolatey.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\chocolatey.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cinst.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cinst.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\clist.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\clist.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cpack.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cpack.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cpush.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cpush.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cuninst.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cuninst.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cup.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cup.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cver.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\cver.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\redirects\RefreshEnv.cmd'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\7z.dll'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\7z.dll.manifest'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\7z.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\7z.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\7z.exe.manifest'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\7zip.license.txt'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\checksum.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\checksum.exe.config'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\checksum.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\checksum.license.txt'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\shimgen.exe'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\shimgen.exe.ignore'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\tools\shimgen.license.txt'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\[Co
ntent_Types].xml'.
VERBOSE: Created
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\pac
kage\services\metadata\core-properties\0f188391ebb44f408b3c77de8dcfdfe0.psmdcp'
.
Installing Chocolatey on the local machine
VERBOSE: Loading module from path
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateysetup.psm1'.
VERBOSE: Exporting function 'Initialize-Chocolatey'.
VERBOSE: Importing function 'Initialize-Chocolatey'.
VERBOSE: Loading module from path
'C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\too
ls\chocolateyInstall\helpers\chocolateyInstaller.psm1'.
Creating ChocolateyInstall as an environment variable (targeting 'Machine')
  Setting ChocolateyInstall to 'C:\ProgramData\chocolatey'
WARNING: It's very likely you will need to close and reopen your shell
  before you can use choco.
Restricting write permissions to Administrators
We are setting up the Chocolatey package repository.
The packages themselves go to 'C:\ProgramData\chocolatey\lib'
  (i.e. C:\ProgramData\chocolatey\lib\yourPackageName).
A shim file for the command line goes to 'C:\ProgramData\chocolatey\bin'
  and points to an executable in 'C:\ProgramData\chocolatey\lib\yourPackageName'.

Creating Chocolatey folders if they do not already exist.

WARNING: You can safely ignore errors related to missing log files when
  upgrading from a version of Chocolatey less than 0.9.9.
  'Batch file could not be found' is also safe to ignore.
  'The system cannot find the file specified' - also safe.
chocolatey.nupkg file not installed in lib.
 Attempting to locate it from bootstrapper.
PATH environment variable does not have C:\ProgramData\chocolatey\bin in it. Adding...
WARNING: Not setting tab completion: Profile file does not exist at
'C:\Users\ContainerAdministrator\Documents\WindowsPowerShell\Microsoft.PowerShe
ll_profile.ps1'.
Chocolatey (choco.exe) is now ready.
You can call choco from anywhere, command line or powershell by typing choco.
Run choco /? for a list of functions.
You may need to shut down and restart powershell and/or consoles
 first prior to using choco.
Ensuring Chocolatey commands are on the path
Ensuring chocolatey.nupkg is in the lib folder
Removing intermediate container 774f8ed5f97b
 ---> ef6c6af3e7fa
Step 5/5 : RUN choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"
 ---> Running in e205047435fa
Chocolatey v0.11.3
Installing the following packages:
git
By installing, you accept licenses for the packages.
Progress: Downloading git.install 2.34.1... 100%
Progress: Downloading chocolatey-core.extension 1.3.5.1... 100%
Progress: Downloading git 2.34.1... 100%

chocolatey-core.extension v1.3.5.1 [Approved]
chocolatey-core.extension package files install completed. Performing other installation steps.
 Installed/updated chocolatey-core extensions.
 The install of chocolatey-core.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'

git.install v2.34.1 [Approved]
git.install package files install completed. Performing other installation steps.
Parameter: no git shell integration
Using Git LFS
Installing 64-bit git.install...
git.install has been installed.
WARNING: Can't find git.install install location
  git.install can be automatically uninstalled.
Environment Vars (like PATH) have changed. Close/reopen your shell to
 see the changes (or in powershell/cmd.exe just type `refreshenv`).
 The install of git.install was successful.
  Software installed to 'C:\Program Files\Git\'

git v2.34.1 [Approved]
git package files install completed. Performing other installation steps.
 The install of git was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

Chocolatey installed 3/3 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Removing intermediate container e205047435fa
 ---> 25818678f9a2
Successfully built 25818678f9a2

I guess that leaves Host OS or Docker version changes as potential culprits for the behaviour (as server 2022 tests were using Microsoft docker distribution).

So, we're trying to blame Windows 11 process isolation as the root cause of this issue?

server 2022 tests were using Microsoft docker distribution

You can use Stevedore so you have 100% identical Docker engine on both client and server Windows.

Ok, I have repeated the test on server 2022 using Stevedore v0.0.2, the test passes, and git installs correctly.

I'm having some problems getting stevedore to install on win11, its erroring out creating a user, both through chocolatey and directly with the msi. I just loggged a bug report here: https://github.com/slonopotamus/stevedore/issues/12.

@TomSmithGR As a quick note, I would find it easier to read if you put the output text inside code-blocks (```-delimited blocks).

TBBIe: thanks for the tip, sorry a bit new to markdown, I will correct the posts.

Ok, I managed to get stevedore working, and repeated the tests. Git still fails on win11 with process isolation, even with the same docker version. It must be something different between windows 11 and server 2022 hosts when using process isolation.

Results of docker build on win11 using stevedore v0.0.2 and process isolation. win11-stevedore.txt

Oh dear, I hope this isn't another ABI breakage issue like we saw back in February 2020, this time caused by some update to Windows 11. Microsoft made a point of promising full ABI compatibility between existing Windows Server 2022 container images and all future versions of Windows Server 2022 and Windows 11, ensuring process isolation mode never breaks.

I spun up a win11 VM in azure to verify this isn't machine specific, and Git will install fine in process isolation there.

Both machines have identical windows builds, both using the same stevedore version, both building the same dockerfile, one succeeds, one fails.

We are now really in the weeds, either its something specific to this box, like hardware fail, corruption of a file somewhere, or its something really weird like a cpu specific issue (im AMD, VM is xeon).

I just updated to Win11 (22000) (after not being able to use process isolation on 19044) and am getting to the exact same error when trying to force ltsc2022 in process isolation: git.install (exited -1073741819):

Operating system:           Windows 10 Pro (Build 22000.376)
Docker daemon version:      20.10.11
NVIDIA Docker supported:    No
Maximum image size:         1000GB
Available disk space:       1.7 TiB
Total system memory:        31.87 GiB physical, 4.75 GiB virtual
Number of processors:       12 physical, 24 logical (AMD Ryzen 9 5900X)

I tried a hyperv-build for ue4-build-prerequisites, and a rerun with process isolation when that was cached locally, but that did not work:

Step 15/27 : RUN .\Engine\Build\BatchFiles\RunUAT.bat BuildGraph -target="Make Installed Build Win64" -script=Engine/Build/InstalledEngineBuild.xml -set:HostPlatformOnly=true -set:WithDDC=true  &&        (if exist C:\UnrealEngine\LocalBuilds\InstalledDDC rmdir /s /q C:\UnrealEngine\LocalBuilds\InstalledDDC) && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
---> Running in 99f822572c72
Running AutomationTool...
C:\BuildTools\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "Ionic.Zip.Reduced". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
C:\BuildTools\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "Microsoft.VisualStudio.Setup.Configuration.Interop". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
Platform\Mac\MacToolChain.cs(11,7): error CS0246: The type or namespace name 'Ionic' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
Platform\Mac\UEDeployMac.cs(10,7): error CS0246: The type or namespace name 'Ionic' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
Platform\TVOS\TVOSToolChain.cs(13,7): error CS0246: The type or namespace name 'Ionic' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
Platform\TVOS\TVOSToolChain.cs(14,7): error CS0246: The type or namespace name 'Ionic' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
Platform\Android\AndroidAARHandler.cs(13,7): error CS0246: The type or namespace name 'Ionic' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
Platform\IOS\IOSToolChain.cs(14,7): error CS0246: The type or namespace name 'Ionic' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
Platform\IOS\IOSToolChain.cs(15,7): error CS0246: The type or namespace name 'Ionic' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
Platform\Windows\UEBuildWindows.cs(11,17): error CS0234: The type or namespace name 'VisualStudio' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj]
RunUAT.bat ERROR: AutomationTool failed to compile.
BUILD FAILED
The command 'cmd /S /C .\Engine\Build\BatchFiles\RunUAT.bat BuildGraph -target="Make Installed Build Win64" -script=Engine/Build/InstalledEngineBuild.xml -set:HostPlatformOnly=true -set:WithDDC=true  &&  (if exist C:\UnrealEngine\LocalBuilds\InstalledDDC rmdir /s /q C:\UnrealEngine\LocalBuilds\InstalledDDC) && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.' returned a non-zero code: 1
[ue4-docker build] Error: failed to build image "adamrehn/ue4-minimal:4.27.2-ltsc2022-vs2017".

So we have a success on a Win11 VM on an Intel Xeon host, and two failures on Win11 bare-metal AMD-based hosts. I was suspicious that it might be CPU-related earlier, but apparently didn't say so here.

So if someone is able to take the time to reproduce the failure with an AMD-based cloud VM to contrast with @TomSmithGR 's success on an Intel-based cloud VM, then we probably have a very clear repro we can kick over to Microsoft to chase up with their own channel partners.

If an AMD-based cloud host running Win11 repros the issue, I'd be interested if it also repros with the AMD host running LTSC2022, as then:

It's something about the AMD CPU and containers, and Win11 is just a red herring due to relative CPU popularity of Intel on Server and AMD on Client.
MS will probably care strongly about this. and it'd live entirely within the Server bailiwick for a change.

ue4-docker build --no-engine --no-full --exclude debug --exclude templates --monitor -basetag ltsc2022 -isolation=process 4.27.2

Azure VM D8as v4 (AMD EPYC 7763)

LTSC2022 : Passing the git.install step ✅ (don't remember if failed any vcredist)
Win11: Passing the git.install step ✅ (but failing vcredist2005, vcredist2008 and vcredist2010)

Annoyingly enough, that Win11 VM is the exact same version as my local machine (which fails):

Kernel Version: 10.0 22000 (22000.1.amd64fre.co_release.210604-1628)
Operating System: Windows 10 Pro Version 2009 (OS Build 22000.376)

It seems I don't have any local Intel machines compatible with Windows 11.

Just so I'm clear, the Dockerfile.txt attached to https://github.com/adamrehn/ue4-docker/issues/216#issuecomment-1005786099 is the repro-case being tested now? So we get the failures independent of ue4-docker?

I have something to add. Although git failing stops the process, its not the only problem. I created exported dockerfiles from ue4docker, then modified them to continue on the git error above. Several other dependencies also errored out during install. The log below is of a trial run. For clarity this is running on windows 11 host, Docker version 20.10.11, build dea9396, using my local threadripper machine.

Trial run with exported dockerfile - process isolation

PS D:\ue4docker\test\ue4-build-prerequisites> docker build . --isolation=process
Sending build context to Docker daemon  13.82kB
Step 1/12 : ARG BASEIMAGE=mcr.microsoft.com/windows/servercore:ltsc2022
Step 2/12 : ARG DLLSRCIMAGE=mcr.microsoft.com/windows/server:ltsc2022
Step 3/12 : FROM ${DLLSRCIMAGE} as dlls
 ---> 4ea596b53499
Step 4/12 : FROM ${BASEIMAGE} as prerequisites
 ---> 136110e1a05c
Step 5/12 : SHELL ["cmd", "/S", "/C"]
 ---> Using cache
 ---> 099f6e4a9b61
Step 6/12 : COPY --from=dlls     C:\Windows\System32\avicap32.dll     C:\Windows\System32\avifil32.dll     C:\Windows\System32\avrt.dll     C:\Windows\System32\d3d10warp.dll     C:\Windows\System32\D3DSCache.dll     C:\Windows\System32\dsound.dll     C:\Windows\System32\dxva2.dll     C:\Windows\System32\glu32.dll     C:\Windows\System32\mf.dll     C:\Windows\System32\mfplat.dll     C:\Windows\System32\mfplay.dll     C:\Windows\System32\mfreadwrite.dll     C:\Windows\System32\msacm32.dll     C:\Windows\System32\msdmo.dll     C:\Windows\System32\msvfw32.dll     C:\Windows\System32\opengl32.dll     C:\Windows\System32\ResampleDMO.dll     C:\Windows\System32\ResourcePolicyClient.dll     C:\Windows\System32\
 ---> Using cache
 ---> 50c522baa80c
Step 7/12 : LABEL com.adamrehn.ue4-docker.sentinel="1"
 ---> Using cache
 ---> aead79f7ba8d
Step 8/12 : RUN reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem /v LongPathsEnabled /t REG_DWORD /d 1 /f && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 3d5ac96ab7d3
Step 9/12 : RUN powershell -NoProfile -ExecutionPolicy Bypass -Command "Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> df394981878a
Step 10/12 : COPY buildtools-exitcode.py install-prerequisites.bat C:\
 ---> Using cache
 ---> 4fb6bd0590a7
Step 11/12 : ARG VISUAL_STUDIO_BUILD_NUMBER
 ---> Using cache
 ---> fdc1aa3d19f3
Step 12/12 : RUN C:\install-prerequisites.bat %VISUAL_STUDIO_BUILD_NUMBER% && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Running in 92698b25e500

C:\>choco install -y git --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'"   ||
Chocolatey v0.11.3
Installing the following packages:
git
By installing, you accept licenses for the packages.
Progress: Downloading git.install 2.34.1... 100%
Progress: Downloading chocolatey-core.extension 1.3.5.1... 100%
Progress: Downloading git 2.34.1... 100%

chocolatey-core.extension v1.3.5.1 [Approved]
chocolatey-core.extension package files install completed. Performing other installation steps.
 Installed/updated chocolatey-core extensions.
 The install of chocolatey-core.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'

git.install v2.34.1 [Approved]
git.install package files install completed. Performing other installation steps.
Parameter: no git shell integration
Using Git LFS
Installing 64-bit git.install...
ERROR: Running ["C:\ProgramData\chocolatey\lib\git.install\tools\Git-2.34.1-64-bit.exe" /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /NOCANCEL /SP- /LOG /COMPONENTS="icons,assoc,assoc_sh,,,,gitlfs,icons\quicklaunch" /o:PathOption=Cmd /o:BashTerminalOption=ConHost /o:CRLFOption=CRLFCommitAsIs ] was not successful. Exit code was '-1073741819'. See log for possible error messages.
The install of git.install was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\git.install\tools\chocolateyInstall.ps1'.
 See log for details.

git v2.34.1 [Approved]
git package files install completed. Performing other installation steps.
 The install of git was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

Chocolatey installed 2/3 packages. 1 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Failures
 - git.install (exited -1073741819) - Error while running 'C:\ProgramData\chocolatey\lib\git.install\tools\chocolateyInstall.ps1'.
 See log for details.

C:\>choco install -y choco-cleaner curl vcredist-all windows-sdk-10-version-1809-windbg   || goto :error
Chocolatey v0.11.3
Installing the following packages:
choco-cleaner;curl;vcredist-all;windows-sdk-10-version-1809-windbg
By installing, you accept licenses for the packages.
Progress: Downloading choco-cleaner 0.0.8.4... 100%

choco-cleaner v0.0.8.4 [Approved]
choco-cleaner package files install completed. Performing other installation steps.
Adding C:\ProgramData\chocolatey\bin\choco-cleaner.bat and pointing it to powershell command C:\tools\BCURRAN3\choco-cleaner.ps1
PATH environment variable does not have C:\tools\BCURRAN3 in it. Adding...
Refreshing environment variables from the registry for powershell.exe. Please wait...
Finished
SUCCESS: The scheduled task "choco-cleaner" has successfully been created.

Folder: \
TaskName                                 Next Run Time          Status
======================================== ====================== ===============
choco-cleaner                            1/9/2022 11:00:00 PM   Ready
Now configured to run Choco-Cleaner at 11:00 PM every SUNDAY.
You can manually run Choco-Cleaner from the Command Prompt, Powershell, or the Windows Start Menu icon.
Environment Vars (like PATH) have changed. Close/reopen your shell to
 see the changes (or in powershell/cmd.exe just type `refreshenv`).
 The install of choco-cleaner was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.
Progress: Downloading curl 7.81.0... 100%

curl v7.81.0 [Approved]
curl package files install completed. Performing other installation steps.
Extracting 64-bit C:\ProgramData\chocolatey\lib\curl\tools\curl-7.81.0-win64-mingw.zip to C:\ProgramData\chocolatey\lib\curl\tools...
C:\ProgramData\chocolatey\lib\curl\tools
 ShimGen has successfully created a shim for curl.exe
 The install of curl was successful.
  Software installed to 'C:\ProgramData\chocolatey\lib\curl\tools'
Progress: Downloading vcredist2005 8.0.50727.619501... 100%
Progress: Downloading vcredist-all 1.0.1... 100%
Progress: Downloading vcredist2008 9.0.30729.616104... 100%
Progress: Downloading vcredist2010 10.0.40219.32503... 100%
Progress: Downloading vcredist2012 11.0.61031... 100%
Progress: Downloading vcredist2013 12.0.40660.20180427... 100%
Progress: Downloading vcredist140 14.30.30704... 100%
Progress: Downloading KB3033929 1.0.5... 100%
Progress: Downloading chocolatey-windowsupdate.extension 1.0.4... 100%
Progress: Downloading KB3035131 1.0.3... 100%
Progress: Downloading KB2919355 1.0.20160915... 100%
Progress: Downloading KB2919442 1.0.20160915... 100%
Progress: Downloading KB2999226 1.0.20181019... 100%
Progress: Downloading vcredist2015 14.0.24215.20170201... 100%
Progress: Downloading vcredist2017 14.16.27033... 100%

vcredist2005 v8.0.50727.619501 [Approved]
vcredist2005 package files install completed. Performing other installation steps.
Downloading vcredist2005 64 bit
  from 'https://download.microsoft.com/download/8/B/4/8B42259F-5D70-43F4-AC2E-4B208FD8D66A/vcredist_x64.EXE'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2005\8.0.50727.619501\vcredist_x64.EXE (3.03 MB).
Download of vcredist_x64.EXE (3.03 MB) completed.
Hashes match.
Installing vcredist2005...
ERROR: Running ["C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2005\8.0.50727.619501\vcredist_x64.EXE" /Q ] was not successful. Exit code was '-2147023829'. See log for possible error messages.
The install of vcredist2005 was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\vcredist2005\tools\chocolateyInstall.ps1'.
 See log for details.

vcredist2008 v9.0.30729.616104 [Approved]
vcredist2008 package files install completed. Performing other installation steps.
Downloading vcredist2008 64 bit
  from 'https://download.microsoft.com/download/5/D/8/5D8C65CB-C849-4025-8E95-C3966CAFD8AE/vcredist_x64.exe'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2008\9.0.30729.616104\vcredist_x64.exe (4.97 MB).
Download of vcredist_x64.exe (4.97 MB) completed.
Hashes match.
Installing vcredist2008...
ERROR: Running ["C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2008\9.0.30729.616104\vcredist_x64.exe" /Q /NORESTART ] was not successful. Exit code was '1'. See log for possible error messages.
The install of vcredist2008 was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\vcredist2008\tools\chocolateyInstall.ps1'.
 See log for details.

vcredist2010 v10.0.40219.32503 [Approved]
vcredist2010 package files install completed. Performing other installation steps.
Downloading vcredist2010 64 bit
  from 'https://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x64.exe'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2010\10.0.40219.32503\vcredist_x64.exe (9.8 MB).
Download of vcredist_x64.exe (9.8 MB) completed.
Hashes match.
Installing vcredist2010...
ERROR: Running ["C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2010\10.0.40219.32503\vcredist_x64.exe" /Q /NORESTART ] was not successful. Exit code was '1'. See log for possible error messages.
The install of vcredist2010 was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\vcredist2010\tools\chocolateyInstall.ps1'.
 See log for details.

vcredist2012 v11.0.61031 [Approved]
vcredist2012 package files install completed. Performing other installation steps.
WARNING: Url has SSL/TLS available, switching to HTTPS for download
Downloading vcredist2012 64 bit
  from 'https://download.microsoft.com/download/1/6/B/16B06F60-3B20-4FF2-B699-5E9B7962F9AE/VSU_4/vcredist_x64.exe'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2012\11.0.61031\vcredist_x64.exe (6.85 MB).
Download of vcredist_x64.exe (6.85 MB) completed.
Hashes match.
Installing vcredist2012...
vcredist2012 has been installed.
WARNING: Url has SSL/TLS available, switching to HTTPS for download
Downloading vcredist2012
  from 'https://download.microsoft.com/download/1/6/B/16B06F60-3B20-4FF2-B699-5E9B7962F9AE/VSU_4/vcredist_x86.exe'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2012\11.0.61031\vcredist_x86.exe (6.25 MB).
Download of vcredist_x86.exe (6.25 MB) completed.
Hashes match.
Installing vcredist2012...
vcredist2012 has been installed.
  vcredist2012 may be able to be automatically uninstalled.
 The install of vcredist2012 was successful.
  Software installed as 'EXE', install location is likely default.

vcredist2013 v12.0.40660.20180427 [Approved]
vcredist2013 package files install completed. Performing other installation steps.
Downloading vcredist2013 64 bit
  from 'https://download.microsoft.com/download/0/5/6/056dcda9-d667-4e27-8001-8a0c6971d6b1/vcredist_x64.exe'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2013\12.0.40660.20180427\vcredist_x64.exe (6.87 MB).
Download of vcredist_x64.exe (6.87 MB) completed.
Hashes match.
Installing vcredist2013...
vcredist2013 has been installed.
Downloading vcredist2013
  from 'https://download.microsoft.com/download/0/5/6/056dcda9-d667-4e27-8001-8a0c6971d6b1/vcredist_x86.exe'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist2013\12.0.40660.20180427\vcredist_x86.exe (6.21 MB).
Download of vcredist_x86.exe (6.21 MB) completed.
Hashes match.
Installing vcredist2013...
vcredist2013 has been installed.
  vcredist2013 may be able to be automatically uninstalled.
 The install of vcredist2013 was successful.
  Software installed as 'exe', install location is likely default.

chocolatey-windowsupdate.extension v1.0.4 [Approved]
chocolatey-windowsupdate.extension package files install completed. Performing other installation steps.
 Installed/updated chocolatey-windowsupdate extensions.
 The install of chocolatey-windowsupdate.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-windowsupdate'

KB3035131 v1.0.3 [Approved]
kb3035131 package files install completed. Performing other installation steps.
Skipping installation because update KB3035131 does not apply to this operating system (Microsoft Windows Server 2022 Datacenter).
 The install of kb3035131 was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

KB3033929 v1.0.5 [Approved]
kb3033929 package files install completed. Performing other installation steps.
Skipping installation because update KB3033929 does not apply to this operating system (Microsoft Windows Server 2022 Datacenter).
 The install of kb3033929 was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

KB2919442 v1.0.20160915 [Approved]
kb2919442 package files install completed. Performing other installation steps.
Skipping installation because this hotfix only applies to Windows 8.1 and Windows Server 2012 R2.
 The install of kb2919442 was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

KB2919355 v1.0.20160915 [Approved]
kb2919355 package files install completed. Performing other installation steps.
Skipping installation because this hotfix only applies to Windows 8.1 and Windows Server 2012 R2.
 The install of kb2919355 was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

KB2999226 v1.0.20181019 [Approved] - Possibly broken
kb2999226 package files install completed. Performing other installation steps.
Skipping installation because update KB2999226 does not apply to this operating system (Microsoft Windows Server 2022 Datacenter).
 The install of kb2999226 was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

vcredist140 v14.30.30704 [Approved]
vcredist140 package files install completed. Performing other installation steps.
Downloading vcredist140-x86
  from 'https://download.visualstudio.microsoft.com/download/pr/2250605e-e48f-43ed-ba6e-e0cc18bc530d/AC75A82D873E6B6F98B1D293042380764D7D263C43438E50D564FA58C9F891C2/VC_redist.x86.exe'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist140\14.30.30704\VC_redist.x86.exe (13.06 MB).
Download of VC_redist.x86.exe (13.06 MB) completed.
Hashes match.
Installing vcredist140-x86...
vcredist140-x86 has been installed.
Downloading vcredist140-x64 64 bit
  from 'https://download.visualstudio.microsoft.com/download/pr/2250605e-e48f-43ed-ba6e-e0cc18bc530d/A9F5D2EAF67BF0DB0178B6552A71C523C707DF0E2CC66C06BFBC08BDC53387E7/VC_redist.x64.exe'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\vcredist140\14.30.30704\VC_redist.x64.exe (24.1 MB).
Download of VC_redist.x64.exe (24.1 MB) completed.
Hashes match.
Installing vcredist140-x64...
vcredist140-x64 has been installed.
  vcredist140 may be able to be automatically uninstalled.
 The install of vcredist140 was successful.
  Software installed as 'exe', install location is likely default.

vcredist2015 v14.0.24215.20170201 [Approved]
vcredist2015 package files install completed. Performing other installation steps.
 The install of vcredist2015 was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

vcredist2017 v14.16.27033 [Approved]
vcredist2017 package files install completed. Performing other installation steps.
 The install of vcredist2017 was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.

vcredist-all v1.0.1 [Approved]
vcredist-all package files install completed. Performing other installation steps.
 The install of vcredist-all was successful.
  Software install location not explicitly set, it could be in package or
  default install location of installer.
Progress: Downloading windows-sdk-10-version-1809-windbg 10.0.17763.132... 100%

windows-sdk-10-version-1809-windbg v10.0.17763.132 [Approved]
windows-sdk-10-version-1809-windbg package files install completed. Performing other installation steps.
Downloading windows-sdk-10-version-1809-windbg
  from 'https://download.microsoft.com/download/5/C/3/5C3770A3-12B4-4DB4-BAE7-99C624EB32AD/windowssdk/winsdksetup.exe'
Progress: 100% - Completed download of C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\windows-sdk-10-version-1809-windbg\10.0.17763.132\winsdksetup.exe (1.29 MB).
Download of winsdksetup.exe (1.29 MB) completed.
Hashes match.
Installing windows-sdk-10-version-1809-windbg...
windows-sdk-10-version-1809-windbg has been installed.
  windows-sdk-10-version-1809-windbg may be able to be automatically uninstalled.
 The install of windows-sdk-10-version-1809-windbg was successful.
  Software installed as 'EXE', install location is likely default.

Chocolatey installed 15/18 packages. 3 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Installed:
 - vcredist2013 v12.0.40660.20180427
 - vcredist140 v14.30.30704
 - vcredist-all v1.0.1
 - kb3033929 v1.0.5
 - curl v7.81.0
 - vcredist2012 v11.0.61031
 - kb3035131 v1.0.3
 - kb2919355 v1.0.20160915
 - kb2919442 v1.0.20160915
 - kb2999226 v1.0.20181019
 - windows-sdk-10-version-1809-windbg v10.0.17763.132
 - vcredist2017 v14.16.27033
 - choco-cleaner v0.0.8.4
 - vcredist2015 v14.0.24215.20170201
 - chocolatey-windowsupdate.extension v1.0.4

Failures
 - vcredist2008 (exited 1) - Error while running 'C:\ProgramData\chocolatey\lib\vcredist2008\tools\chocolateyInstall.ps1'.
 See log for details.
 - vcredist2010 (exited 1) - Error while running 'C:\ProgramData\chocolatey\lib\vcredist2010\tools\chocolateyInstall.ps1'.
 See log for details.
 - vcredist2005 (exited -2147023829) - Error while running 'C:\ProgramData\chocolatey\lib\vcredist2005\tools\chocolateyInstall.ps1'.
 See log for details.
The command 'cmd /S /C C:\install-prerequisites.bat %VISUAL_STUDIO_BUILD_NUMBER% && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.' returned a non-zero code: 1
PS D:\ue4docker\test\ue4-build-prerequisites>

I'm currently testing this on a VirtualBox VM running on top of Ryzen 3700X, stay tuned.

Okay, I've reproed this issue on VirtualBox 6.1.28 running on top of Ryzen 3700X.

Install Windows 11. SHA256 of Win11_English_x64v1.iso I used: 4bc6c7e7c61af4b5d1b086c5d279947357cff45c2f82021bb58628c2503eb64e. I did not install any Windows updates.
Install Docker engine. For example, via: 2.1. Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1')) in an elevated PowerShell 2.2. choco install -y stevedore in the same elevated PowerShell
Reboot
Create directory somewhere

Create Dockerfile in that directory with the following contents:

FROM mcr.microsoft.com/windows/servercore:ltsc2022
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'Continue'; $verbosePreference='Continue';"]
ENV CHOCO_URL=https://chocolatey.org/install.ps1
RUN Set-ExecutionPolicy Bypass -Scope Process -Force; \
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; \
iex ((New-Object System.Net.WebClient).DownloadString("$env:CHOCO_URL"));
RUN choco install -y vcredist-all

docker build --isolation=process . in that directory. Expected: it succeeds. Actual: lots of stuff fails with Exit code was '-2147023829'

We now can conclude that this issue is 100% independent of ue4-docker.

ATTENTION: I am NOT reproducing the issue if I use RUN choco install -y --params "'/GitOnlyOnPath /NoAutoCrlf /WindowsTerminal /NoShellIntegration /NoCredentialManager'" git command in Dockerfile instead of RUN choco install -y vcredist-all.

So, did anyone report this to Microsoft?

I didn't, I only put a ticket in with chocolatey, and that didn't go anywhere.

Looking back at this ticket, there's actually two error codes going on. The original report from the git installer was -1073741819, i.e. 0xc0000005 aka STATUS_ACCESS_VIOLATION (per the Microsoft Error Lookup Tool).

This one doesn't seem to repro on VMs:

https://github.com/adamrehn/ue4-docker/issues/216#issuecomment-1007262775
https://github.com/adamrehn/ue4-docker/issues/216#issuecomment-1007367186
~~https://github.com/adamrehn/ue4-docker/issues/216#issuecomment-1006392114~~ Edit: Oops, this was an Intel-based VM.

but repro'd for two users on bare-metal:

The vcredist installers are failing with -2147023829, i.e. 0x8007042b aka ERROR_PROCESS_ABORTED, on both VMs and bare-metal.

https://github.com/adamrehn/ue4-docker/issues/216#issuecomment-1007328270 is the only instance of this one on bare-metal, because @TomSmithGR ran the vcredist install by hand after showing the git failure, while all the other bare-metal tests stopped at the git install failure. So I'm calling that consistent for now.

That suggests we're seeing two different issues here, or the same issue in multiple ways, and the VM-ness of the host affects only one of those ways. The fact that this affects the process-isolation container but not the out-of-container environment points me at a kernel/userspace ABI mismatch issue between LTSC2022 and Windows 11 that is only triggered by AMD CPUs. This suggests that an LTSC2022 host would not show this problem, as the user-space would be the same between the host and the container.

Looking back, I don't see a result for an with ltsc2022 host, either in a VM or bare-metal. Sadly, the one test I can see on ltsc2022 in a VM only noted that the access violation failed to repro, which is consistent with all the other (later) VM-based tests, but didn't record whether the aborted process failure occurred in that setup.

To wildly speculate a little, I wonder if there's actually a bug in the LTSC2022 userspace code calling some kernel API, and it generates out-of-bounds memory access that was fine in the LTSC2022 kernel, but the Windows 11 kernel catches and kills the process in response. Given it's a hardware difference, it would have to be in a hardware-implemented like the NX bit, or a microcode spectre mitigation or similar. That might also explain why one of the issues is VM-sensitive, if the Hypervisor has prevented the CPU-level catch in one case, but not the other.

I don't see a result for an with ltsc2022 host

I have successfully fully built Unreal Engine on ltsc2022 when was working on https://github.com/adamrehn/ue4-docker/pull/199

I've also had success with ltsc2022 on an Azure Intel host (after disabling HW Network acceleration), but the question I failed to answer above was whether it works on ltsc2022 on Azure AMD EPYC VM.

Awesome, so that is consistent with the "ABI back-compat has an issue" guess. (Sorry, I did mean "ltsc2022 AMD host", we haven't seen any repro's on Intel hosts that I'm aware of).

Quick extra note: I've seen both Zen 2 (3700X) and Zen 3 (5900X and EPYC 7763) architecture in the reports, so it's not necessarily a new-CPU issue. There was a Windows 11 issue that affected all these CPUs that has now been fixed.

More WAG: I wonder if VBS, HVCI, or MBEC are involved here? They're not CPU-specific features, but are probably have CPU-specific implementations if they work in concert with hardware-level features. It looks like VBS and HVCI can be toggled (and are controlled separately inside VMs), but MBEC just comes with HVCI on supported CPUs (Zen 2 or later in the AMD case). And it seems that MBEC is disabled by default in VMs.

I'm gonna retest ltsc2022 + AMD CPU to make sure that only Windows 11 is affected.

@TomSmithGR could you also say what is your CPU model where you observe the issue please?

I'm gonna retest ltsc2022 + AMD CPU to make sure that only Windows 11 is affected.

So. Exactly the same setup as in https://github.com/adamrehn/ue4-docker/issues/216#issuecomment-1007367186 (same VirtualBox 6.1.28, same Ryzen 3700X, same Docker engine, same process isolation) with the only difference: I now use Windows Server 2022 LTSC instead of Windows 11. And I do NOT reproduce neither Exit code was '-2147023829' nor Exit code was '-1073741819'.

My verdict: Windows 11 process isolation is broken.

Do we know Process Isolation has been stable on Windows 10? I am seeing similar errors (although different exit codes) trying to build images on an old server (Xeon X5680) Windows 10 Pro (Build 19041.1288)

Do we know Process Isolation has been stable on Windows 10?

Yes. Exactly the same setup as in https://github.com/adamrehn/ue4-docker/issues/216#issuecomment-1007367186 (same VirtualBox 6.1.28, same Ryzen 3700X, same Docker engine, same process isolation) with the only difference: I now use fully updated Windows 10 LTSC 2019. I do NOT reproduce neither Exit code was '-2147023829' nor Exit code was '-1073741819'.

And overall, I've built LOTS of ue4-docker Windows images during last year on different Windows 10 machines.

So, Microsoft has somewhat confirmed that Win11 process isolation is broken. There's nothing we can do here, except maybe using Hyper-V isolation by default when we see a Win11 host.

Oh, wait, we already use Hyper-V isolation on Win11 by default.

If and when a fix is issued, we can reject versions we know are affected. I'm assuming it'll turn out to be a host-system issue; if it turns out to be a container-image issue, that'll be harder to detect without more image introspection that we currently do.

Aaand... Let's reopen. I'm observing this on Windows 10 LTSC 2019, but only on AMD CPU.

This means two things:

This issue is not specific to Windows 11
Something has changed. I used exactly the same VM lots of times in 2021 and it just worked.

The error code observed in #230 is 0xC0000409 STATUS_STACK_BUFFER_OVERRUN. Per https://devblogs.microsoft.com/oldnewthing/20190108-00/?p=100655 and https://docs.microsoft.com/en-us/cpp/intrinsics/fastfail?view=msvc-170 there might be a more-interesting error code buried in there. It also could be a genuine stack buffer overrun, of course.

I'm not sure how to extract such a 'more interesting' error code though.

Also, for "exactly the same VM", have security updates been applied since it last worked?

have security updates been applied since it last worked?

Yes. And docker images (windowsservercore and windows) are also newer.

Okay, so it's possible Windows patches have introduced the problem in either the container host or container image. >_<

Since we're talking process isolation, that means that if this is the same issue in both Windows 11 and Server 2019, a change was made in either both Server 2019 and Windows 11, or both the Server 2019 and Server 2022 container images.

In my opinion, the latter seems more likely. And should be easier to test, since old base images aren't removed from MCR and can be named by full-version tag. So if you happen to have a record of a base tag that worked previously, you could try that precise same container image against the newer host in your VM. If it fails, then the remaining validation of a host-side change requires rolling back the host version (or building a new VM, I guess) and only updating to a known-to-work-previously update.

Copying from #230:

~I'm reproing this on Windows 10 LTSC 2019 build 17763.316 host + 10.0.17763.2686 image.~

Current hypothesis: this issue is caused by mcr.microsoft.com/windows/servercore image updates. Trying older versions.

~10.0.17763.1935 FAIL~ ~10.0.17763.2237 FAIL~

These tests are invalid, see next comment.

Although this symptom is different, I'd suggest testing with both host and container on the same side of the February 11 2020 security update, which introduced some kind of ABI breakage between host and container.

I know the ABI breakage definitely affected ue4-docker because some of the stuff Chocolatey installs is using 32-bit installers, and I happened to get caught by this issue trying to build newer container images on an older host. But that was on an Intel machine, so perhaps the AMD machines will report that same breakage with the error we're seeing here.

So for Server 2019, that's both host and container either ≤973, or host ≥ 1039 and container ≥ 1040.

Oh maaaan, I forgot about that thing.

This severely complicates things. I am not sure how I upgrade host to what I had in spring/summer of 2021 (that's when I was building tons of ue4-build-prerequisites while working on #144 and #187) and not any further.

You might be able to use https://www.powershellgallery.com/packages/PSWindowsUpdate/2.2.0.2 to update to a KB from the update list. I've not tried this though.

You could also try just grabbing the cumulative update package from the Microsoft Update Catalog (the link's under "How to get this update") and work out how to install that .msu file. I think you can right-click and hit install like a .msi file, but I might be wrong about that.

New plan: I'm fully updating my 2019 host system and re-testing everything :D That Feb 2020 thing could have corrupted all of my tests I did on AMD CPU in #230.

10.0.17763.2686 Windows 10 LTSC 2019 host (fully updated as of today) + 10.0.17763.2686 Docker image (this is the one you get by ltsc2019 tag as of today) + process isolation = everything works.

@alexgeek WRT your 20H2 issue in #230. 20H2 is going EOL on 2022-05-10, in just a month and a half. I don't think it worth trying to fix anything on it. Just let it go and switch to a more long-term options.

Currently supported and functional systems with process isolation:

Windows Server 2019 + ltsc2019 image
Windows 10 LTSC 2019 + ltsc2019 image
Windows Server 2022 + ltsc2022 image
Windows 11 + ltsc2022 image but only on Intel CPUs

Technically, Windows Server 2016 and Windows 10 LTSB 2016 would also work with ltsc2016 image, but ue4-docker has dropped support for them in https://github.com/adamrehn/ue4-docker/pull/187 (though you might have success with ue4-docker versions older than v0.0.89).

So is my only option for Windows 10 Pro to use LTSC 2019 without process isolation?

So either I move to Win 11 or Windows Server?

Yup. I followed up on #232 about this part specifically.

adamrehn / ue4-docker