search

adamrehn / ue4-docker

Windows and Linux containers for Unreal Engine 4
https://docs.adamrehn.com/ue4-docker/
MIT License
779 stars 172 forks source link

VS2019 build tools are installed, rather than VS2017 as documented #26

Closed TBBle closed 5 years ago

TBBle commented 5 years ago

This is probably an upstream (Microsoft) issue; I was bitten by the failure with GenerateProjectFiles.bat with VS 2019, despite believing I was using VS2017.

The build failure is because VS2019 changed the relative path to current MSBuild.exe, and the UE4 build scripts were falling back to the 4.0 version is part of the older Framework SDK, but cannot compile UE4's C# code. It's already been fixed in UE4 upstream. If you also had VS2017 installed, it would find that version of MSBuild and work correctly.

From what I can see, the Docker script is correctly pulling the VS2017 Build Tools installer (https://aka.ms/vs/15/release/vs_buildtools.exe for VS2017 contrasting with https://aka.ms/vs/16/release/vs_buildtools.exe for VS2019). So this is probably an issue with the upstream installer either having been accidentally updated to VS2019, or mis-set expectations of what the installer does.

It's possible the upstream installer is installing the VS2017 toolchain, but applies the VS2019 MSBuild.exe path change. That would be awful. >_<

adamrehn commented 5 years ago

Damn, it's even worse than that. I'm looking at the filesystem from a fresh build of the ue4-build-prerequisites image, and not only has it installed MSBuild in the new VS2019 location, it doesn't appear to have installed the Build Tools themselves at all, either the VS2017 or VS2019 version. I'll continue investigating and post any findings here.

adamrehn commented 5 years ago

Progress update:

adamrehn commented 5 years ago

As expected, MSBuild is installed to the new location, so I've simply copied it into the VS2017 location. (The only files that get overwritten are two DLLs in the Bin\Roslyn subdirectory, and the versions that overwrite them look to be the same as the files they're replacing.) I've pushed the patch up in commit 2ff06d1. Once I've tested it to make sure it works with 4.19 through to 4.22, I'll update the PyPI package and push the new build(s) of the ue4-build-prerequisites image to Docker Hub, addressing issue #25.

TBBle commented 5 years ago

For the record, UE's bug tracker for the moved MSBuild fix is https://issues.unrealengine.com/issue/UE-72173

Overall, VS2019 with the VS2017 compiler toolchain installed seems reasonable, that's the part UBT actually uses, the IDE version is incidental.

The visualstudio.15.chman definitely lists

{"id":"Microsoft.VisualStudio.Product.BuildTools","version":"15.9.28307.586","type":"ChannelProduct",

so it's annoying and disappointing that it seems to be ignoring it.

Thank you for your fast turnaround.

Edit: At some point, we may need to be able to control the compiler used, as VS2019/UE support matures, and teams start migrating. It sounds like currently the mapping of compiler to UE version is static?

adamrehn commented 5 years ago

Yeah, at the moment the Windows images use the VS2015 toolchain for UE4.19 and the VS2017 toolchain for everything else. Engine version detection is performed at runtime during the build process, since users can specify an arbitrary git repo and branch to clone for the ue4-source image, which prevents ue4-docker from making version assumptions at the start of the build process.

At some point in the future it'll likely become necessary to maintain multiple Windows base images (VS2015 toolchain, VS2017 toolchain, VS2019 toolchain, etc.) and have the user specify which base image to use when performing a build. I've been trying to keep things as automatic as possible so far, but the scalability issues inherent in the current approach have become more and more evident with each new release of UE4.

adamrehn commented 5 years ago

Update: unfortunately the new version of MSBuild doesn't appear to work correctly with Unreal Engine 4.21, even when copied into the correct location:

Running AutomationTool...
C:\BuildTools\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(1661,5): error MSB4036: The "GetReferenceNearestTargetFrameworkTask" task was not found. Check the following: 1.) The name of the task in the project file is the same as the name of the task class. 2.) The task class is "public" and implements the Microsoft.Build.Framework.ITask interface. 3.) The task is correctly declared with <UsingTask> in the project file, or in the *.tasks files located in the "C:\BuildTools\MSBuild\15.0\Bin" directory. [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\AutomationTool.csproj]
RunUAT.bat ERROR: AutomationTool failed to compile.
BUILD FAILED

I'm going to investigate options for controlling the versions of components that get installed, in the hopes that the VS2017 version of MSBuild can be installed somehow.

adamrehn commented 5 years ago

So it turns out I was on the right track using the VS2017 channel manifest after all, I just needed to add the flag --channelId VisualStudio.15.Release to force the installer to use that channel. Inspecting an image built after commit 7940d85 confirms that the VS2017 Build Tools are now being installed once more:

C:\Program Files (x86)\Microsoft Visual Studio\Installer>vswhere -all -products *
Visual Studio Locator version 2.6.7+91f4c1d09e [query version 2.0.2250.60958]
Copyright (C) Microsoft Corporation. All rights reserved.

instanceId: d27b35da
installDate: 6/4/2019 12:27:09 AM
installationName: VisualStudio/15.9.11+28307.586
installationPath: C:\BuildTools
installationVersion: 15.9.28307.586
productId: Microsoft.VisualStudio.Product.BuildTools
productPath: C:\BuildTools\Common7\Tools\LaunchDevCmd.bat
state: 4294967295
isComplete: 1
isLaunchable: 1
isPrerelease: 0
isRebootRequired: 0
displayName: Visual Studio Build Tools 2017
description: The Visual Studio Build Tools allows you to build native and managed MSBuild-based applications without requiring the Visual Studio IDE. There are options to install the Visual C++ compi
lers and libraries, MFC, ATL, and C++/CLI support.
channelId: VisualStudio.15.Release
channelUri: C:\Users\ContainerAdministrator\AppData\Local\Temp\visualstudio.15.release.chman
enginePath: C:\Program Files (x86)\Microsoft Visual Studio\Installer\resources\app\ServiceHub\Services\Microsoft.VisualStudio.Setup.Service
installChannelUri: C:\Users\ContainerAdministrator\AppData\Local\Temp\visualstudio.15.release.chman
releaseNotes: https://go.microsoft.com/fwlink/?LinkId=660692#15.9.11
thirdPartyNotices: https://go.microsoft.com/fwlink/?LinkId=660708
updateDate: 2019-04-05T14:07:09.0247689Z
catalog_buildBranch: d15.9
catalog_buildVersion: 15.9.28307.586
catalog_id: VisualStudio/15.9.11+28307.586
catalog_localBuild: build-lab
catalog_manifestName: VisualStudio
catalog_manifestType: installer
catalog_productDisplayVersion: 15.9.11
catalog_productLine: Dev15
catalog_productLineVersion: 2017
catalog_productMilestone: RTW
catalog_productMilestoneIsPreRelease: False
catalog_productName: Visual Studio
catalog_productPatchVersion: 11
catalog_productPreReleaseMilestoneSuffix: 1.0
catalog_productRelease: RTW
catalog_productSemanticVersion: 15.9.11+28307.586
catalog_requiredEngineVersion: 1.18.1049.33485
properties_campaignId:
properties_channelManifestId: VisualStudio.15.Release/15.9.11+28307.586
properties_nickname:
properties_setupEngineFilePath: C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installershell.exe

Frustratingly however, the VS2019 version of MSBuild is still being installed (there is no Bin subdirectory under the 15.0 folder, only the Current folder):

C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin>MSBuild /?
Microsoft (R) Build Engine version 16.0.461+g6ff56ef63c for .NET Framework

Edit: I've just realised that I forgot to remove the line rmdir /S /Q C:\BuildTools\MSBuild\15.0\Bin from the next stage in the script, so I'm re-running the build with this commented out so I can see what's actually being placed in that Bin folder when it's not being inadvertently removed.

adamrehn commented 5 years ago

Success! After 9907c58:

C:\BuildTools\MSBuild\15.0\Bin>MSBuild /?
Microsoft (R) Build Engine version 15.9.21+g9802d43bc3 for .NET Framework

It turns out the version of MSBuild under C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools was a red herring, since this executable is actually already present in the latest version of the microsoft/dotnet-framework:4.7.2-sdk base image and was not being created by the Build Tools installer at all.

TBBle commented 5 years ago

The presence of the 2019 Build Tools MSBuild executable might prove a problem for UE4 4.22.1, as the hotfix they applied (per UE-72173) will prefer that version.

adamrehn commented 5 years ago

@TBBle fortunately this doesn't appear to be the case, as the VS2019 MSBuild executable is not reported by the vswhere query that UE4 uses to locate the toolchain.

I've now tested the latest updates with 4.19.2, 4.20.3, 4.21.2, and 4.22.0, and everything appears to be working correctly. The fixes are now live in ue4-docker version 0.0.27.