Ensure all Windows build machines have same compiler level

[pshipton]

Platform: Windows

Architecture: any

@jdekonin mentioned the Windows playbooks install the latest VS2017 compiler. Meaning that if build machines are created at different times, they could have different levels of the VS2017 compiler on them. Ideally all build machines should have the same level of VS2017, so the build isn't affected by what machine it lands on.

Not sure if the same issue applies to VS2010.

[sxa]

Do you have a proposal for how to implement this? And is this a theoretical issue or something you've seen an issue with?

[pshipton]

My proposal is to fix at some known version for all machines, until such time as all machines are updated to a different known version. This a theoretical issue. If there is some difference in the code generated from different compiles, it could cause a lot of confusion if the behavior changes based on the machine where the build was compiled.

[sxa]

My question was more whether you knew of an easy way to fix it - we pull the installer from https://aka.ms/vs/15/release/vs_community.exe

We might be able (subject to licensing) to copy a fixed version of that executable somewhere and run that instead of downloading it from the original source each time, but that would also be subject to whether MS's installer allows the downloading of earlier versions.

[pshipton]

I believe @jdekonin mentioned there is a way to get particular versions.

[jdekonin]

Not for getting the older versions. I have @kevin-bonilla looking into whether that is possible.

From my msdn account I don't see any download for older versions, so unless there is a method via the installer or someone has an "offline installer installation", I don't see any options to accomplish this yet.

[ghost]

Not for getting the older versions. I have @kevin-bonilla looking into whether that is possible.

From my msdn account I don't see any download for older versions, so unless there is a method via the installer or someone has an "offline installer installation", I don't see any options to accomplish this yet.

Yeah I still need to do some further research to see if this is still possible. I tried digging through some DevOps microsoft tool as well called Visual Studio Dev Essentials but that product as well only gives me the Visual Studio Installer which would install the latest 15.9.22

[sxa]

Copying in my comment from #1354 as I don't want conversations fragmented across issues This would need wider approval I think before doing it than just me. Are we also confident that such a solution is allowed within the terms of the license agreement for the free compiler download?

[pshipton]

It's possible to create an offline cache of a particular version. Then all machines should get that version until there is a decision to switch to a different one. https://docs.microsoft.com/en-us/visualstudio/install/create-an-offline-installation-of-visual-studio?view=vs-2017

[pshipton]

I believe all the OpenJ9 machines were updated to 15.9.22, @jdekonin will need to confirm. However the latest version is now 15.9.23 so you can't get 15.9.22 any more unless you already have an offline cache for it.

[jdekonin]

OpenJ9 machines are at 15.9.22 using a offline installer for VS Pro, not the Community version as corporate machines can't be using the free version.

[pshipton]

@jdekonin The OpenJ9 machines used publicly at https://ci.eclipse.org/openj9?

[jdekonin]

Yes these machines are also using the same VS Pro. I checked with my licensing contact and she advised we could use the same for internal IBM and the IBM Cloud Windows instances.

This was more of a convenience issue as my team supports not only these 6 windows machines but an internal IBM machine build/test farm that cannot use the VS Community version.

[sxa]

Here at adopt we use the community version as installed by the playbooks. Unless this can be done through the community version this /may/ be a non-starter ... Although we do have some MSFT people on the team but I have a slight preference for using the same freely available tools that others may use our scripts with.

[pshipton]

The community version is listed in the instructions for creating an offline cache. https://docs.microsoft.com/en-us/visualstudio/install/create-an-offline-installation-of-visual-studio?view=vs-2017

[sxa]

Bumping this as from a secure (SSDF) perspective and from a perspective of trying to recover MSVS_2013 this would be beneficial

[andrew-m-leonard]

This is a potential problem for "reproducible builds", as any difference in compiler, even a minor build version, introduces binary differences... Without the exact same build level of VisualStudio, you cannot reproduce a build.

@gdams fyi

[gdams]

@andrew-m-leonard take a look at how it's done at the OpenJDK project (https://github.com/openjdk/jdk/blob/87340fd5408d89d9343541ff4fcabde83548a598/.github/workflows/build-windows.yml#L100)

Essentially we need to set a specific version of msvc toolset and we should then have identical compiler levels on all windows machines.

[andrew-m-leonard]

So i've done some test installs and builds, and if I install vs_Community19, then select say build tools 14.27, it seems to also install the latest 14.29 if I do a straight build. However, I can select 14.27 directly to build openjdk by using the --with-msvc-toolset-version=14.27 configure arg. Which is good, however it does not allow you to select "fix update levels", eg.14.27.111, which is on most of the Adoptium windows nodes currently. On a new node if you select 14.27 you get 14.27.120 currently.

For reproducibility it seems to imply we will need to cache the install ?

[andrew-m-leonard]

@gdams this seems problematic, I am guessing Microsoft applies fix updates to previous versions, and the vs_installer will only download the latest "fix update"? So although we want say "14.27.111", it will only install the current "14.27.120".

[sxa]

@andrew-m-leonard To confirm here, you were getting different (non-reproducible) output even when ubilding with different fix updates right?

[andrew-m-leonard]

@andrew-m-leonard To confirm here, you were getting different (non-reproducible) output even when ubilding with different fix updates right?

@sxa correct

[andrew-m-leonard]

Maybe as part of provisioning a new MSVC "update" to all Windows machines we create a network offline copy of the exact same version: https://docs.microsoft.com/en-us/visualstudio/install/create-a-network-installation-of-visual-studio?view=vs-2022 This can then be used on another Windows box to install the exact same level.

[andrew-m-leonard]

@sxa do we have any infrastructure "archive" space where we could potentially store caches of "MSVC network installer layouts". Which could then be remotely accessed maybe from an ansible task to setup a new Windows node?

[sxa]

Potentially (we already do so for gcc), although as mentioned elsewhere we need to determine if it can be a public space or if that would have any problems relating to the product license.

[andrew-m-leonard]

Potentially (we already do so for gcc), although as mentioned elsewhere we need to determine if it can be a public space or if that would have any problems relating to the product license.

Yeah, I think the Scope of License section probably prohibits that: https://visualstudio.microsoft.com/license-terms/mlt031819/

[sxa]

We can do it regardless, but we'll need to make the playbooks account for the situation where they can't access our "secret" cached copies (EIther by doing what it does now or requiring the user to install one manually outside the playbooks...)

Based on that document, will a fixed version of the bootstrapper be adequate? I REALLY want to avoid having a windows machine dedicated to hosting VS versions if I can avoid it.

And have you verified that it can work with the community edition which we use?

[gdams]

What about if we built a devkit for the Windows build using the scripts (https://github.com/openjdk/jdk/blob/master/make/devkit/createWindowsDevkit.sh) and then pull that down to the machine for builds? Would that create a consistent compiler?

[sxa]

@andrew-m-leonard How comfortable are we that we have a solution here now, and should we look at deploying this onto the build machines? This would, of course, have implications as to the reproducibility of our existing JDK17/19 releases if the compiler version is switched to a fixed level.

[andrew-m-leonard]

What about if we built a devkit for the Windows build using the scripts (https://github.com/openjdk/jdk/blob/master/make/devkit/createWindowsDevkit.sh) and then pull that down to the machine for builds? Would that create a consistent compiler?

That looks like it essentially copies the relevant compiler bits to a "DevKit" folder for your own usage, as long as that place is on your private infra, as per the MSVC license. So it's essentially a cut down "MSVC install layer". If storage is an issue for where we put a "MSVC install layer" image (~9Gb), then we could try adopting this script.

[andrew-m-leonard]

@andrew-m-leonard How comfortable are we that we have a solution here now, and should we look at deploying this onto the build machines? This would, of course, have implications as to the reproducibility of our existing JDK17/19 releases if the compiler version is switched to a fixed level.

@sxa Yes, using a fixed MSVC layer install does work.

[sxa]

@andrew-m-leonard What's your feeling on trying to switch the older JDK releases over to building on Windows Server 2022 before the release later this month?

[andrew-m-leonard]

So I guess we need to do this at some point, and can't foresee there being a problem. The only thing to question, is capacity of 2022 nodes for all versions? or do we just provision via Azure?

[sxa]

The intention world be to start switching all of the other build machines we have over to 2022 if we make the change. We're potentially building the versions at once for any given version (x32/x64/Arm64) and we have two available at the moment: https://ci.adoptium.net/label/build&&win2022/

[sxa]

Done for JDK20+ by moving to server 2022 build machines - we should look to progress this with earlier versions after the July releases

[sxa]

@steelhead31 has confirmed that the wix label can be applied to the win2022 nodes and still produce usable installers so we may now aim to simply reprovision some of the older ones with win2022 and the fixed compiler instead of adjusting them.

[sxa]

We currently have 1/5 online machine labelled build&win2012

Noting that JDK8 and perhaps more are still built using those labels so they are currently all being funnelled to that one machine so it is currently a single point of failure (Although we can bring others back online if required)

[steelhead31]

As per this issue #3158 and this PR adoptium/ci-jenkins-pipelines#771 all Temurin JDK builds have been migrated to Windows Server 2022. These machines were all built using the infrastructure playbook, and as such have a consistent Visual Studio 2017 layout applied to them.