chore: use docker bake to generate docker compose file

[lemeurherve]

This PR uses docker bake to generate the content of the docker compose files for Windows images from the same docker-bake.hcl used for Linux images.

Fixes:

• 426

The JSON output of docker buildx bake windows --print is converted with yq into the YAML format expected by docker compose to build the corresponding images. (We can't use bake for building as it's not functional on Windows yet.)

~The docker compose files are generated (here) from a Linux agent as it avoids tedious yq queries escaping for PowerShell and installing docker buildx on Windows agents.~

EDIT: As docker buildx bake windows --print can be used on Windows, make isn't required for contributors. And as yq is already required, the only new requirement for contributors is https://github.com/docker/buildx included in Docker Desktop, one of the most common docker installation types on Windows.

Example of resulting build-windows.yaml (note that no arg depends on any variable)

```yaml services: nanoserver-1809_jdk11: image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk11 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: "1809" tags: - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk11 - docker.io/jenkins/ssh-agent:nanoserver-1809-jdk11 nanoserver-1809_jdk17: image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk17 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: "1809" tags: - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk17 - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809 - docker.io/jenkins/ssh-agent:nanoserver-1809 - docker.io/jenkins/ssh-agent:nanoserver-1809-jdk17 nanoserver-1809_jdk21: image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk21 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: "1809" tags: - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk21 - docker.io/jenkins/ssh-agent:nanoserver-1809-jdk21 nanoserver-ltsc2019_jdk11: image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk11 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk11 - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk11 nanoserver-ltsc2019_jdk17: image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk17 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk17 - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019 - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019 - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk17 nanoserver-ltsc2019_jdk21: image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk21 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk21 - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk21 nanoserver-ltsc2022_jdk11: image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk11 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk11 - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk11 nanoserver-ltsc2022_jdk17: image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk17 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk17 - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022 - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022 - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk17 nanoserver-ltsc2022_jdk21: image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk21 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk21 - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk21 windowsservercore-ltsc2019_jdk11: image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk11 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk11 - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk11 windowsservercore-ltsc2019_jdk17: image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk17 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk17 - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019 - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019 - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk17 windowsservercore-ltsc2019_jdk21: image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk21 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk21 - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk21 windowsservercore-ltsc2022_jdk11: image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk11 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk11 - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk11 windowsservercore-ltsc2022_jdk17: image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk17 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk17 - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022 - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022 - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk17 windowsservercore-ltsc2022_jdk21: image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk21 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk21 - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk21 ``` Obtained with: ```console ON_TAG=true VERSION=1.2.3 docker buildx bake -f docker-bake.hcl windows --print \ | yq -P '.target[] | del(.platforms) | {(. | key): {"image": .tags[0], "build": .}}' | yq '{"services": .}' ``` Note: this command works for Linux/MacOS, Windows (or more specifically PowerShell) requires backslashes in front of every double quote and setting environment variables on their own.

Enhancements:

• Regroup all Linux and Windows image definitions in a common docker-bake.hcl file containing every parameters and helper functions
• Remove build-windows.yaml
• Simplify Jenkinsfile
• Simplify build.ps1 by not having to massage parameters to deal with the fact that there are no mcr.microsoft.com/powershell ltsc2019 images, only 1809 ones
• Simplify updatecli manifests
• Prepare the terrain for when docker buildx will be functional to build Windows images

Images content remains untouched hence the "chore" label.

Testing done

Local tests and CI.

Submitter checklist

• [x] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• [x] Ensure that the pull request title represents the desired changelog entry
• [x] Please describe what you did
• [x] Link to relevant issues in GitHub or Jira
• [x] Link to relevant pull requests, esp. upstream and downstream changes
• [x] Ensure you have provided tests - that demonstrates feature works or fixes the issue

[lemeurherve]

Could you explain what is the "problem" you want to solve with this PR? As much as it looks interesting and fun technically, I fail to see the gain

The main problem I have with the current status is that we don't have a "pure" definition of Windows images with the docker compose file as it is, we need to massage data with build.ps1 in order to obtain the correct value for TOOLS_WINDOWS_VERSION for example.

This PR proposes having "pure" images definitions, inside a single common source of truth (the bake file), containing all the helper functions (without having to replicate them in build.ps1) and all the parameters needed to generate the images.

It also paves the way for when docker build bake will be usable to build Windows images.

[lemeurherve]

In draft while I'm extracting unrelated changes in another PR and improve the contributor doc.

Main goals of this PR:

• Single source of thrust for the definitions of linux and windows images in the bake file
• Simplify build.ps1, now almost only used to manage tests as all calls to docker can be done in a single line
• Pave the way for building windows image with bake, we won't need docker compose anymore

[lemeurherve]

It looks like you need a lot of additional tools if you are on Windows to be able to build the image (including GNU make and yq): am I correct?

I reworked the PR to generate the docker compose file from a Windows agent instead of a Linux one, make isn't required on Windows anymore.

As yq is already required, the only new requirement is https://github.com/docker/buildx, added to the agents with https://github.com/jenkins-infra/packer-images/pull/1215.

There are a lot of tiny improvement which looks really useful without any doubts, such as factorizations to variables in the HCL => do you mind splitting into PR with different goals to make it easier for othger maintainers to review please (I'm not willing to give and approval with so much changes at the same time).

Extracted in:

• 419

• 425

• 424

• 421

Could you explain what is the "problem" you want to solve with this PR? As much as it looks interesting and fun technically, I fail to see the gain

I've detailed everything in #426.

Only blocker for now is the missing updated documentation for windows contributor: this PR introduces way more requirements for them and it must be documented.

Currently working on it, the rest is in https://github.com/jenkinsci/docker-ssh-agent/pull/415/commits/7bcc98f061ebc4965c82ac6a5828920220784993

I'll pass the PR in ready for review when I'll add the documentation.

[lemeurherve]

Pull request ready for review, this commit especially: https://github.com/jenkinsci/docker-ssh-agent/pull/415/commits/f93da6a9c7f169d600a5434d8648667a8411be57?w=1

The rest are cherry-picks from the other pull requests, amongst those #419 and #424 are required to be merged before merging this one.

Or eventually, this PR can also be merged as it is, superseding all extracted PRs if they're OK for maintainers.

[lemeurherve]

~As I put the docker compose file generation into a function, I realized that moving Pester initialization to its own function would result in a more readable build.ps1 code~ ~(can be extracted to its own PR if deemed preferable)~

Resulting build.ps1 "main" script

https://github.com/jenkinsci/docker-ssh-agent/blob/e93d536859c8d4435042b59aa4d3f538915f3b2f/build.ps1#L186-L258

EDIT: droped, can't make it work on ci.jenkins.io while it works locally, will try later.

[lemeurherve]

As the cherry-picked commits and the corresponding (merged) pull requests have diverged, I'm rebasing this pull request into a clean commit.