search

adoptium / ci-jenkins-pipelines

jenkins pipeline build scripts
Apache License 2.0
19 stars 71 forks source link

Update MS VS ReDist DLLs distributed to "latest stable" #1104

Closed andrew-m-leonard closed 1 month ago

andrew-m-leonard commented 2 months ago

Recent issue highlighted we currently ship an old UCRT DLL that can cause in-compatibilities with certain API calls with the latest Win SDK. We should move up to a more recent "fixed" stable set, we should also specify the complete set so we are consistent: Currently this, with the additional DLLs loaded from the installed VS layout:

--with-ucrt-dll-dir='C:/progra~2/wi3cf2~1/10/Redist/10.0.22621.0/ucrt/DLLs/x64'

Should update to use the complete set:

--with-msvcr-dll: points to the VS C runtime main DLL
--with-vcruntime-1-dll: points to the VS C runtime extra DLL
--with-msvcp-dll: points to the VS C++ runtime main DLL
--with-ucrt-dll-dir: point to the directory for the Universal C runtime DLLs (all of them)
andrew-m-leonard commented 1 month ago

Possibly update to latest: image

andrew-m-leonard commented 1 month ago

Latest versions from Microsoft: https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

andrew-m-leonard commented 1 month ago

The full historic list of MSVC++ version are here: https://en.wikipedia.org/wiki/Microsoft_Visual_C++ The current is:

Visual Studio 2022 version 17.10.1  17.10.1     2022    1940    194033811   14.40.33811
andrew-m-leonard commented 1 month ago

The latest MSVC Redist files can be downloaded as installer executable file here: https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

When vc_redist.x64.exe will install the redist DLLs into the c:\Windows\System32 directory as long as there isn't a later version of the redist in that folder already.

To package a set of redist DLL's necessary to package with Temurin, I would suggest the following process:

  1. Determine the set of redist files that are packaged with Temurin
  2. On a suitable Windows VM ensure the c:\Windows\System32 directory has all the redist DLLs removed
  3. Then download the required redist version from https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170
  4. Run the redist.exe, which will extract the DLLs into the c:\Windows\System32 directory
  5. Zip the necessary redist DLLs into a zip file and store...

For building Temurin at ci.adoptium.net:

  1. Extract the "stored" redist files into a suitable local directory
  2. Configure the openjdk make with necessary configure options pointing at the extract redist local directory: 
    --with-msvcr-dll: points to the VS C runtime main DLL
--with-vcruntime-1-dll: points to the VS C runtime extra DLL
--with-msvcp-dll: points to the VS C++ runtime main DLL
--with-ucrt-dll-dir: point to the directory for the Universal C runtime DLLs (all of them)
andrew-m-leonard commented 1 month ago

@sxa @steelhead31 Your thoughts please, if we manually package the latest "redist" DLLs, where could we put them for our build nodes?

steelhead31 commented 1 month ago

@andrew-m-leonard I think we could, package up the zip files , and have them deployed by Ansible into a specifie folder on the windows build machines, we could approach this in a way to allow multiple versions, or alternatively just a single version... something along the lines of..

c:\temurin-msvc-redist\version1.7.0 c:\temurin-msvc-redist\version2.7.0 c:\temurin-msvc-redist\latest ..etc..etc..

Could then define the locations for the parameters listed per version as required ?

andrew-m-leonard commented 1 month ago

@andrew-m-leonard I think we could, package up the zip files , and have them deployed by Ansible into a specifie folder on the windows build machines, we could approach this in a way to allow multiple versions, or alternatively just a single version... something along the lines of..

c:\temurin-msvc-redist\version1.7.0 c:\temurin-msvc-redist\version2.7.0 c:\temurin-msvc-redist\latest ..etc..etc..

Could then define the locations for the parameters listed per version as required ?

Thanks Scott, yes that sounds reasonable, I will create the latest Redist version package (17.10 - 14.40.33810)

andrew-m-leonard commented 1 month ago

The Microsoft Redist "update" package installer seems to only install the vcruntime, msvcp.dll and ucrtbase.dll, all of which are put into c:\Windows\System32 when you run say the vc_Redist_x64.exe. It does not install the api-ms-crt*.dll's which seem only present in the Windows Kit/Redist/ucrt/DLLs folder.

Trying to accurately create our own Redist.zip packaging x64,arm64,x86 and all the necessary dlls is going to be tricky, and probably not the right thing to do.

We should probably instead create a "Windows DevKit" for VS2022 using: https://github.com/openjdk/jdk/blob/master/make/devkit/createWindowsDevkit.sh This requires the "required" versions of VS2022 and Windows Kit SDK to be installed on your machine, it will then create a "DevKit" package from them.

We should thus look to update to the latest(possibly LTSC) MS VS2022 update from: https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-history and create a "Windows DevKit" from it.

andrew-m-leonard commented 1 month ago

Creating and using a updated Windows VS2022 "DevKit" would then resolve issue: https://github.com/adoptium/temurin-build/issues/3887

andrew-m-leonard commented 1 month ago

A Windows DevKit would require further investigation and testing. To alleviate the user issues found by https://github.com/adoptium/temurin-build/issues/3887 I am going to try a build test using the latest VS2022 17.10 Redist for just the C and C++ "runtime" DLLs:

--with-msvcr-dll: points to the VS C runtime main DLL
--with-vcruntime-1-dll: points to the VS C runtime extra DLL
--with-msvcp-dll: points to the VS C++ runtime main DLL
fthevenet commented 1 month ago

Hi Andrew

In order to get the UCRT dll to redistribute with the jdk, it seems we have to get them from the Windows SDK: https://devblogs.microsoft.com/cppblog/introducing-the-universal-crt/#distributing-software-that-uses-the-universal-crt (see point 6: "App-local deployment of the Universal CRT is supported")

The "easiest" way to extract a complete collection for these DLLs, is to proceed as follow:

  1. Download an ISO image of the latest SDK from this location: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/
  2. Expand the ISO image to disk
  3. Locate the installation bundle "Universal CRT Redistributable-x86_en-us.msi" and de-compile it using the "dark.exe" MSI de-compiler that comes with the Wix Toolkit. I.e. 
    c:\Tools\wix311\bin\dark.exe -v -o c:\sdk\out\ucrt.wxs  "c:\sdk\26100.1742.240904-1906.ge_release_svc_prod1_WindowsSDK\Installers\Universal CRT Redistributable-x86_en-us.msi" -x c:\sdk\out
  4. The DLLs binaries will have been extracted in "C:\sdk\out\File", but the files are named using the their msi IDs
  5. The Wix sources generated by the de-compiler in c:\sdk\out\ucrt.wxs contains the correspondence between each ID and the dll names, so it should be easy to write a script that renames them.

~While this isn't exactly "easy" (I said "easiest"...), doing things this way presents several advantages over having to run the installer on an (hopefully) clean machine and fish out what we believe are the freshly installed files, mainly that I find it easier to trust we have the extracted the right files. It might also be simpler to automate the process.~

EDIT: Actually, unlike the VC_redist bundle for the legacy C runtime which dumps everything in system32, running "Universal CRT Redistributable-x86_en-us.msi" installed the DLL in an easily identifiable location in C:\Program Files (x86)\Windows Kits\10\Redist\, and it does not overwrite previous versions. So afterall, it might be simpler to just get the files directly from there and avoid having to deal with dark magic.

fthevenet commented 1 month ago

By the way, you can use dark.exe in a similar way to extract the legacy VC runtime DLLs from the VC_redist bundle.

andrew-m-leonard commented 1 month ago

Hi Andrew

In order to get the UCRT dll to redistribute with the jdk, it seems we have to get them from the Windows SDK: https://devblogs.microsoft.com/cppblog/introducing-the-universal-crt/#distributing-software-that-uses-the-universal-crt (see point 6: "App-local deployment of the Universal CRT is supported")

The "easiest" way to extract a complete collection for these DLLs, is to proceed as follow:

1. Download an ISO image of the latest SDK from this location: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/

2. Expand the ISO image to disk

3. Locate the installation bundle "Universal CRT Redistributable-x86_en-us.msi" and de-compile it using the "dark.exe" MSI de-compiler that comes with the Wix Toolkit. I.e.
c:\Tools\wix311\bin\dark.exe -v -o c:\sdk\out\ucrt.wxs  "c:\sdk\26100.1742.240904-1906.ge_release_svc_prod1_WindowsSDK\Installers\Universal CRT Redistributable-x86_en-us.msi" -x c:\sdk\out
4. The DLLs binaries will have been extracted in "C:\sdk\out\File", but the files are named using the their msi IDs

5. The Wix sources generated by the de-compiler in c:\sdk\out\ucrt.wxs  contains the correspondence between each ID and the dll names, so it should be easy to write a script that renames them.

~While this isn't exactly "easy" (I said "easiest"...), doing things this way presents several advantages over having to run the installer on an (hopefully) clean machine and fish out what we believe are the freshly installed files, mainly that I find it easier to trust we have the extracted the right files. It might also be simpler to automate the process.~

EDIT: Actually, unlike the VC_redist bundle for the legacy C runtime which dumps everything in system21, running "Universal CRT Redistributable-x86_en-us.msi" installed the DLL in an easily identifiable location in C:\Program Files (x86)\Windows Kits\10\Redist, and it does not overwrite previous versions. So afterall, it might be simpler to just get the files directly from there and avoid having to deal with dark magic.

Thanks @fthevenet, for the methods. For the linked issue we believe we just need to update the C runtime DLLs only. so I am running a test with just those at the moment

andrew-m-leonard commented 1 month ago

To extract the required Redist DLLs, the safest and easiest way is to locally install an "all architecture" installation as follows:

  1. Determine the "Build Tools" version install bootstrapper required and download from here: https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-history#fixed-version-bootstrappers
  2. Determine the required version of the "Windows SDK" from here: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/
    • Note the version, eg."10.0.26100"
  3. Install locally on a Windows x64 VM using the following command: 
    ./vs_BuildTools.exe --passive --norestart --wait --arch all --add "Microsoft.VisualStudio.Workload.NativeDesktop;includeRecommended;includeOptional" --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.VC.Tools.ARM64 --add Microsoft.VisualStudio.Component.VC.ATL.ARM64 --add Microsoft.VisualStudio.Component.VC.MFC.ARM64
  4. Run the installer manually again to install the required "Windows 11 SDK": 
    ./vs_BuildTools.exe
    • Select "Individual components"
    • Find the required "Windows 11 SDK (REQUIRED VERSION from step 2)"

Now extract the required "Redist" DLLs as follows:

  1. Find the correct MSVC Redist folders, it should be located under folder: 
    C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Redist\MSVC\xx.yy.zzzzz\<arch>\Microsoft.VC143.CRT
  2. Copy the following MSVC Redist DLLs for each architecture (arm64, x64, x86): 
    vcruntime140.dll
vcruntime140_1.dll (not x86)
msvcp140.dll
  3. Find the correct Windows Kit UCRT folder, it should be located under folder: 
    C:\Program Files (x86)\Windows Kits\10\Redist\10.0.xxxxx.0\ucrt
  4. Copy the entire "ucrt" sub-folder containing the DLLs/(arm64, x64, x86), eg: 
    cp -r "Program Files (x86)/Windows Kits/10/Redist/10.0.26100.0/ucrt" .
andrew-m-leonard commented 1 month ago

I have created a Redist update using the above method, for the following versions:

The zip has been uploaded to the devkit_binaries : https://github.com/adoptium/devkit-binaries/releases/tag/vs2022_redist_14.40.33807_10.0.26100.0

andrew-m-leonard commented 1 month ago

Test builds running:

x64: jdk8u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-temurin/514/

jdk11u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-temurin/404/ - Success jdk17u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-windows-x64-temurin/531/ - Success jdk21u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-windows-x64-temurin/165/ - Success jdk23u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk23u/job/jdk23u-windows-x64-temurin/1/ - Success

x86: jdk8u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x86-32-temurin/491/

jdk11u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x86-32-temurin/375/

jdk17u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-windows-x86-32-temurin/363/

aarch64: jdk21u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-windows-aarch64-temurin/3/

jdk23u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk23u/job/jdk23u-windows-aarch64-temurin/1/

andrew-m-leonard commented 1 month ago

Re-running these 3 failures with a longer timeout of x50:

TEST: java/lang/Thread/virtual/stress/PingPong.java#sq
TEST: java/lang/Thread/virtual/stress/PingPong.java#ltq
TEST: java/util/concurrent/BlockingQueue/MultipleProducersSingleConsumerLoops.java

https://ci.adoptium.net/job/Grinder/11135/ Failed, PingPong only managed 235k iterations out of the 10,000,000 default

Re-running with just TEST: java/util/concurrent/BlockingQueue/MultipleProducersSingleConsumerLoops.java : https://ci.adoptium.net/job/Grinder/11136/ - Success 10mins

andrew-m-leonard commented 1 month ago

Mauve related tests that nearly always fail in jdk8u:

09:46:25    MiniMix_5m
09:46:25    MiniMix_10m
09:46:25    MiniMix_aot_5m_0
09:45:30    MauveSingleThrdLoad_HS_5m
09:45:30    MauveSingleInvocLoad_HS_5m
09:45:30    MauveMultiThrdLoad_5m
andrew-m-leonard commented 1 month ago

Newly added hotspot_tier1_runtime_0 and hotspot_tier1_serviceability_0 (https://github.com/adoptium/aqa-tests/pull/5648), that always fail:

hotspot_tier1_runtime:

TEST: runtime/CDSCompressedKPtrs/XShareAuto.java
TEST: runtime/cds/TestCDSVMCrash.java
TEST: runtime/cds/CheckDefaultArchiveFile.java
TEST: runtime/ErrorHandling/UncaughtNativeExceptionTest.java
TEST: runtime/ErrorHandling/MachCodeFramesInErrorFile.java
TEST: runtime/memory/ReadFromNoaccessArea.java
TEST: runtime/jni/nativeStack/TestNativeStack.java

hotspot_tier1_serviceability:

TEST: serviceability/sa/ClhsdbFindPC.java#no-xcomp-core
TEST: serviceability/sa/ClhsdbDumpheap.java
TEST: serviceability/sa/ClhsdbFindPC.java#no-xcomp-process
TEST: serviceability/sa/ClhsdbFindPC.java#xcomp-core
TEST: serviceability/sa/ClhsdbCDSJstackPrintAll.java
TEST: serviceability/sa/ClhsdbJdis.java
TEST: serviceability/sa/ClhsdbJstack.java#id1
TEST: serviceability/sa/ClhsdbPmap.java#core
TEST: serviceability/sa/ClhsdbPrintAs.java
TEST: serviceability/sa/ClhsdbPstack.java#core
TEST: serviceability/sa/ClhsdbSource.java
TEST: serviceability/sa/ClhsdbThreadContext.java
TEST: serviceability/sa/ClhsdbWhere.java
TEST: serviceability/sa/TestJhsdbJstackLock.java
TEST: serviceability/sa/TestClhsdbJstackLock.java
TEST: serviceability/jvmti/RedefineClasses/RedefineSharedClass.java

Raised: https://github.com/adoptium/aqa-tests/issues/5671

andrew-m-leonard commented 1 month ago

Resolved by https://github.com/adoptium/temurin-build/pull/3981