search

adamrehn / ue4-docker

Windows and Linux containers for Unreal Engine 4
https://docs.adamrehn.com/ue4-docker/
MIT License
777 stars 172 forks source link

WinDBG package 404s on choco install #92

Closed highlyunavailable closed 3 years ago

highlyunavailable commented 4 years ago

Output of the ue4-docker info command:

ue4-docker version:         0.0.70 (latest available version is 0.0.70)
Operating system:           Windows 10 Enterprise Version 2004 (OS Build 19041.388)
Docker daemon version:      19.03.12
NVIDIA Docker supported:    No
Maximum image size:         200GB
Available disk space:       308.19 GiB
Total system memory:        31.92 GiB physical, 36.67 GiB virtual
Number of processors:       8 physical, 8 logical

The windbg choco package is broken (see comments on package and thus I can't build a prereq image.

ue4-docker build 4.25.3 --no-engine --no-minimal --no-full

[ue4-docker build] WINDOWS CONTAINER SETTINGS
[ue4-docker build] Isolation mode:               process
[ue4-docker build] Base OS image tag:            2004 (host OS is Windows 10 version 2004)
[ue4-docker build] Memory limit:                 No limit
[ue4-docker build] Detected max image size:      200GB
[ue4-docker build] Directory to copy DLLs from:  C:\WINDOWS\Sysnative

[ue4-docker build] GENERAL SETTINGS
[ue4-docker build] Not excluding any Engine components.

Retrieving the Git credentials that will be used to clone the UE4 repo
Username: 
Password:

[ue4-docker build] Building image "adamrehn/ue4-build-prerequisites:2004"...
Sending build context to Docker daemon  2.085MB
Step 1/27 : ARG BASEIMAGE
Step 2/27 : FROM ${BASEIMAGE} AS dlls
 ---> 99564489cc9c
Step 3/27 : SHELL ["cmd", "/S", "/C"]
 ---> Using cache
 ---> 23a9d3df341d
Step 4/27 : LABEL com.adamrehn.ue4-docker.sentinel="1"
 ---> Using cache
 ---> 5c72dbbad75e
Step 5/27 : RUN mkdir C:\GatheredDlls && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 33e079b1ee65
Step 6/27 : RUN powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 786bb3c4184d
Step 7/27 : RUN choco install -y 7zip curl && choco install -y python --version=3.7.5 && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> d9f362230634
Step 8/27 : COPY *.dll C:\GatheredDlls\
 ---> Using cache
 ---> 8a9e0d244806
Step 9/27 : ARG HOST_VERSION
 ---> Using cache
 ---> 2be6e0fbf580
Step 10/27 : RUN pip install pywin32 && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 517637a304ed
Step 11/27 : COPY copy.py verify-host-dlls.py C:\
 ---> Using cache
 ---> 3efd4ac61efe
Step 12/27 : RUN C:\copy.py "C:\GatheredDlls\*.dll" C:\Windows\System32\ && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 7574029b3983
Step 13/27 : RUN python C:\verify-host-dlls.py %HOST_VERSION% C:\GatheredDlls && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> af01d1cb3a55
Step 14/27 : RUN curl --progress -L "https://download.microsoft.com/download/8/4/A/84A35BF1-DAFE-4AE8-82AF-AD2AE20B6B14/directx_Jun2010_redist.exe" --output %TEMP%\directx_redist.exe && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 563e0a5ec6bf
Step 15/27 : RUN start /wait %TEMP%\directx_redist.exe /Q /T:%TEMP% &&  expand %TEMP%\APR2007_xinput_x64.cab -F:xinput1_3.dll C:\GatheredDlls\ &&       expand %TEMP%\Jun2010_D3DCompiler_43_x64.cab -F:D3DCompiler_43.dll C:\GatheredDlls\ &&
expand %TEMP%\Feb2010_X3DAudio_x64.cab -F:X3DAudio1_7.dll C:\GatheredDlls\ &&   expand %TEMP%\Jun2010_XAudio_x64.cab -F:XAPOFX1_5.dll C:\GatheredDlls\ &&       expand %TEMP%\Jun2010_XAudio_x64.cab -F:XAudio2_7.dll C:\GatheredDlls\ && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> f7bd842f2c32
Step 16/27 : RUN curl --progress -L "https://sdk.lunarg.com/sdk/download/latest/windows/vulkan-runtime-components.zip?u=" --output %TEMP%\vulkan-runtime-components.zip && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 989fb53a1b38
Step 17/27 : RUN 7z e %TEMP%\vulkan-runtime-components.zip -oC:\GatheredDlls -y "*\x64\vulkan-1.dll" && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 45ca755dcc34
Step 18/27 : RUN choco install -y windbg && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Running in a282eb0daaa6
Chocolatey v0.10.15
Installing the following packages:
windbg
By installing you accept licenses for the packages.
Progress: Downloading windbg 10.0.10586.15... 100%

windbg v10.0.10586.15 [Approved]
windbg package files install completed. Performing other installation steps.
Attempt to get headers for http://download.microsoft.com/download/6/A/2/6A2ECE81-C934-4E47-91CC-52DA00A65345/Installers/e8bc712abeffd7c9711ee3f55d4aa99b.cab failed.
  The remote file either doesn't exist, is unauthorized, or is forbidden for url 'http://download.microsoft.com/download/6/A/2/6A2ECE81-C934-4E47-91CC-52DA00A65345/Installers/e8bc712abeffd7c9711ee3f55d4aa99b.cab'. Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (404) Not Found."
Downloading windbg
  from 'http://download.microsoft.com/download/6/A/2/6A2ECE81-C934-4E47-91CC-52DA00A65345/Installers/e8bc712abeffd7c9711ee3f55d4aa99b.cab'
ERROR: The remote file either doesn't exist, is unauthorized, or is forbidden for url 'http://download.microsoft.com/download/6/A/2/6A2ECE81-C934-4E47-91CC-52DA00A65345/Installers/e8bc712abeffd7c9711ee3f55d4aa99b.cab'. Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (404) Not Found."
This package is likely not broken for licensed users - see https://chocolatey.org/docs/features-private-cdn.
The install of windbg was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\windbg\tools\chocolateyInstall.ps1'.
 See log for details.

Chocolatey installed 0/1 packages. 1 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Failures
 - windbg (exited 404) - Error while running 'C:\ProgramData\chocolatey\lib\windbg\tools\chocolateyInstall.ps1'.
 See log for details.

Enjoy using Chocolatey? Explore more amazing features to take your
experience to the next level at
 https://chocolatey.org/compare
The command 'cmd /S /C choco install -y windbg && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.' returned a non-zero code: 404
[ue4-docker build] Error: failed to build image "adamrehn/ue4-build-prerequisites:2004".
PaulR7 commented 4 years ago

Hello,

I had the same issue.

It seems that windbg is not being maintain anymore. I used windows-sdk-10-version-2004-windbg instead, and it works fine for building the image (ue4-build-prerequisites:2004).

adamrehn commented 4 years ago

Apologies for the delay in responding to this issue, I've been inordinately busy over the past couple of months and I'm only now making my way through my GitHub backlog.

Thanks for reporting this, I'll investigate the use of the alternate Chocolatey package and update ue4-docker ASAP.

adamrehn commented 4 years ago

Okay, preliminary testing in a Windows Server Core version 1809 container confirms that the newer SDK package (windows-sdk-10-version-2004-windbg) installs successfully, so mercifully it looks like it won't be necessary to install different SDK versions under different versions of Windows. I've updated the Dockerfile to use the new Chocolatey package in commit 55eaf4c. Once I've performed tests across various versions of Windows and the Unreal Engine to confirm that there are no incompatible configurations then I'll tag a release with the changes.

highlyunavailable commented 4 years ago

That's awesome. Thanks for your work on this!

Agendum commented 3 years ago

Can anybody clarify what the work-around is? Yesterday I just did a pip install ue4-docker so I have the latest version, but sounds like I do not have the fix. When I do a ue4-docker build 4.25.3 --no-engine I get the download error from the original post. Maybe is there a way to apply the fix to my locally installed ue4-docker?

Thanks

Agendum commented 3 years ago

I just want to demonstrate I do have the latest version:

C:\Python39\Scripts>ue4-docker info
ue4-docker version:         0.0.71 (latest available version is 0.0.71)
Operating system:           Windows 10 Enterprise Version 2004 (OS Build 19041.572)
Docker daemon version:      19.03.13
NVIDIA Docker supported:    No
Maximum image size:         200GB
Available disk space:       225.6 GiB
Total system memory:        31.91 GiB physical, 36.66 GiB virtual
Number of processors:       6 physical, 12 logical

Yet ue4-docker build still doesn't yield the fix:

[ue4-docker build] Building image "adamrehn/ue4-build-prerequisites:2004"...
...
Step 18/27 : RUN choco install -y windbg && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Running in 067b601916c1
Chocolatey v0.10.15
Installing the following packages:
windbg
By installing you accept licenses for the packages.
Progress: Downloading windbg 10.0.10586.15... 100%

windbg v10.0.10586.15 [Approved] - Likely broken for FOSS users (due to download location changes)
windbg package files install completed. Performing other installation steps.
Attempt to get headers for http://download.microsoft.com/download/6/A/2/6A2ECE81-C934-4E47-91CC-52DA00A65345/Installers/e8bc712abeffd7c9711ee3f55d4aa99b.cab failed.
  The remote file either doesn't exist, is unauthorized, or is forbidden for url 'http://download.microsoft.com/download/6/A/2/6A2ECE81-C934-4E47-91CC-52DA00A65345/Installers/e8bc712abeffd7c9711ee3f55d4aa99b.cab'. Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (404) Not Found."
Downloading windbg
  from 'http://download.microsoft.com/download/6/A/2/6A2ECE81-C934-4E47-91CC-52DA00A65345/Installers/e8bc712abeffd7c9711ee3f55d4aa99b.cab'
ERROR: The remote file either doesn't exist, is unauthorized, or is forbidden for url 'http://download.microsoft.com/download/6/A/2/6A2ECE81-C934-4E47-91CC-52DA00A65345/Installers/e8bc712abeffd7c9711ee3f55d4aa99b.cab'. Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (404) Not Found."
This package is likely not broken for licensed users - see https://chocolatey.org/docs/features-private-cdn.
The install of windbg was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\windbg\tools\chocolateyInstall.ps1'.
 See log for details.

I mean, I see the fix in master: https://github.com/adamrehn/ue4-docker/blob/master/ue4docker/dockerfiles/ue4-build-prerequisites/windows/Dockerfile#L41

And I see the latest version (0.0.71) was released around the time of this fix: https://pypi.org/project/ue4-docker/#history

Any hints on why this still doesn't work?

highlyunavailable commented 3 years ago

That's because the latest release doesn't have that commit in it. You can check by going to the file under the release tag:

https://github.com/adamrehn/ue4-docker/blob/v0.0.71/ue4docker/dockerfiles/ue4-build-prerequisites/windows/Dockerfile#L41

That said, if you can find that dockerfile in your install, you likely could change that line to match the one in the master branch. For me it's located at C:\Python38\Lib\site-packages\ue4docker\dockerfiles\ue4-build-prerequisites\windows but obviously the directory may be different due to python/virtualenv locations. Use python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())" to find your lib path.

Agendum commented 3 years ago

Brilliant, thank you! Here are the steps I used to get unblocked.

  1. Run the command python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
  2. Change to the printed directory
  3. Open the file ue4docker\dockerfiles\ue4-build-prerequisites\windows\Dockerfile from that directory.
  4. Manually apply the change shown at https://github.com/adamrehn/ue4-docker/commit/55eaf4c635e55dcf57bc0c405fd225d1bbef7acd

Thanks!

TBBle commented 3 years ago

@adamrehn Can we get an 0.0.72 release with 55eaf4c635e55dcf57bc0c405fd225d1bbef7acd please to close this issue?

Or is there some blocking issue or concern with this change that is holding it back?

adamrehn commented 3 years ago

Apologies for the delay in getting back to this issue, I've been drowning in work over the past few months and watching in horror as my GitHub backlog steadily balloons to ever greater sizes.

I had been planning on thoroughly testing this change across a range of configurations (Windows version / UE4 version) to ensure it doesn't break anything, but given that Windows builds are completely broken without it, I guess it's better to tag a release now and identify any edge cases later. The fix is now live in ue4-docker version 0.0.72.