Use only docker-bake.hcl for both Linux and Windows image definitions

[lemeurherve]

What feature do you want to see added?

I propose to have a unique source of truth for all image definitions (Linux and Windows), variables and helper functions.

Goals:

• Avoid mistakes, oversights, inconsistencies or more work to do thanks to unique location of image definitions
• Simplify build.ps1, which won't have to massage data for images definitions anymore
• Pave the way for when we'll be able to use docker buildx bake to build and publish images on Windows

How?

Even if docker buildx bake can't be used for building images on Windows yet, it can already be used on Windows with the --print parameter, outputting Windows images definitions as a JSON.

docker buildx bake windows --print

```json { "group": { "default": { "targets": [ "windows" ] }, "nanoserver": { "targets": [ "nanoserver-1809_jdk11", "nanoserver-1809_jdk17", "nanoserver-1809_jdk21", "nanoserver-ltsc2019_jdk11", "nanoserver-ltsc2019_jdk17", "nanoserver-ltsc2019_jdk21", "nanoserver-ltsc2022_jdk11", "nanoserver-ltsc2022_jdk17", "nanoserver-ltsc2022_jdk21" ] }, "windows": { "targets": [ "nanoserver", "windowsservercore" ] }, "windowsservercore": { "targets": [ "windowsservercore-ltsc2019_jdk11", "windowsservercore-ltsc2019_jdk17", "windowsservercore-ltsc2019_jdk21", "windowsservercore-ltsc2022_jdk11", "windowsservercore-ltsc2022_jdk17", "windowsservercore-ltsc2022_jdk21" ] } }, "target": { "nanoserver-1809_jdk11": { "context": ".", "dockerfile": "windows/nanoserver/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-11", "JAVA_VERSION": "11.0.23+9", "TOOLS_WINDOWS_VERSION": "1809", "WINDOWS_VERSION_TAG": "1809" }, "tags": [ "docker.io/jenkins/ssh-agent:nanoserver-1809-jdk11" ], "platforms": [ "windows/amd64" ] }, "nanoserver-1809_jdk17": { "context": ".", "dockerfile": "windows/nanoserver/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-17", "JAVA_VERSION": "17.0.11+9", "TOOLS_WINDOWS_VERSION": "1809", "WINDOWS_VERSION_TAG": "1809" }, "tags": [ "docker.io/jenkins/ssh-agent:nanoserver-1809-jdk17" ], "platforms": [ "windows/amd64" ] }, "nanoserver-1809_jdk21": { "context": ".", "dockerfile": "windows/nanoserver/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-21", "JAVA_VERSION": "21.0.3+9", "TOOLS_WINDOWS_VERSION": "1809", "WINDOWS_VERSION_TAG": "1809" }, "tags": [ "docker.io/jenkins/ssh-agent:nanoserver-1809-jdk21" ], "platforms": [ "windows/amd64" ] }, "nanoserver-ltsc2019_jdk11": { "context": ".", "dockerfile": "windows/nanoserver/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-11", "JAVA_VERSION": "11.0.23+9", "TOOLS_WINDOWS_VERSION": "1809", "WINDOWS_VERSION_TAG": "ltsc2019" }, "tags": [ "docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk11" ], "platforms": [ "windows/amd64" ] }, "nanoserver-ltsc2019_jdk17": { "context": ".", "dockerfile": "windows/nanoserver/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-17", "JAVA_VERSION": "17.0.11+9", "TOOLS_WINDOWS_VERSION": "1809", "WINDOWS_VERSION_TAG": "ltsc2019" }, "tags": [ "docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk17" ], "platforms": [ "windows/amd64" ] }, "nanoserver-ltsc2019_jdk21": { "context": ".", "dockerfile": "windows/nanoserver/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-21", "JAVA_VERSION": "21.0.3+9", "TOOLS_WINDOWS_VERSION": "1809", "WINDOWS_VERSION_TAG": "ltsc2019" }, "tags": [ "docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk21" ], "platforms": [ "windows/amd64" ] }, "nanoserver-ltsc2022_jdk11": { "context": ".", "dockerfile": "windows/nanoserver/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-11", "JAVA_VERSION": "11.0.23+9", "TOOLS_WINDOWS_VERSION": "ltsc2022", "WINDOWS_VERSION_TAG": "ltsc2022" }, "tags": [ "docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk11" ], "platforms": [ "windows/amd64" ] }, "nanoserver-ltsc2022_jdk17": { "context": ".", "dockerfile": "windows/nanoserver/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-17", "JAVA_VERSION": "17.0.11+9", "TOOLS_WINDOWS_VERSION": "ltsc2022", "WINDOWS_VERSION_TAG": "ltsc2022" }, "tags": [ "docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk17" ], "platforms": [ "windows/amd64" ] }, "nanoserver-ltsc2022_jdk21": { "context": ".", "dockerfile": "windows/nanoserver/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-21", "JAVA_VERSION": "21.0.3+9", "TOOLS_WINDOWS_VERSION": "ltsc2022", "WINDOWS_VERSION_TAG": "ltsc2022" }, "tags": [ "docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk21" ], "platforms": [ "windows/amd64" ] }, "windowsservercore-ltsc2019_jdk11": { "context": ".", "dockerfile": "windows/windowsservercore/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-11", "JAVA_VERSION": "11.0.23+9", "TOOLS_WINDOWS_VERSION": "1809", "WINDOWS_VERSION_TAG": "ltsc2019" }, "tags": [ "docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk11" ], "platforms": [ "windows/amd64" ] }, "windowsservercore-ltsc2019_jdk17": { "context": ".", "dockerfile": "windows/windowsservercore/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-17", "JAVA_VERSION": "17.0.11+9", "TOOLS_WINDOWS_VERSION": "1809", "WINDOWS_VERSION_TAG": "ltsc2019" }, "tags": [ "docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk17" ], "platforms": [ "windows/amd64" ] }, "windowsservercore-ltsc2019_jdk21": { "context": ".", "dockerfile": "windows/windowsservercore/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-21", "JAVA_VERSION": "21.0.3+9", "TOOLS_WINDOWS_VERSION": "1809", "WINDOWS_VERSION_TAG": "ltsc2019" }, "tags": [ "docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk21" ], "platforms": [ "windows/amd64" ] }, "windowsservercore-ltsc2022_jdk11": { "context": ".", "dockerfile": "windows/windowsservercore/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-11", "JAVA_VERSION": "11.0.23+9", "TOOLS_WINDOWS_VERSION": "ltsc2022", "WINDOWS_VERSION_TAG": "ltsc2022" }, "tags": [ "docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk11" ], "platforms": [ "windows/amd64" ] }, "windowsservercore-ltsc2022_jdk17": { "context": ".", "dockerfile": "windows/windowsservercore/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-17", "JAVA_VERSION": "17.0.11+9", "TOOLS_WINDOWS_VERSION": "ltsc2022", "WINDOWS_VERSION_TAG": "ltsc2022" }, "tags": [ "docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk17" ], "platforms": [ "windows/amd64" ] }, "windowsservercore-ltsc2022_jdk21": { "context": ".", "dockerfile": "windows/windowsservercore/Dockerfile", "args": { "JAVA_HOME": "C:/openjdk-21", "JAVA_VERSION": "21.0.3+9", "TOOLS_WINDOWS_VERSION": "ltsc2022", "WINDOWS_VERSION_TAG": "ltsc2022" }, "tags": [ "docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk21" ], "platforms": [ "windows/amd64" ] } } } ```

This output can then be converted with yq (already required for building Windows images) into the format expected by docker compose, still used (for now) for building and publishing Windows images.

Corresponding docker compose file content

```yaml services: nanoserver-1809_jdk11: image: docker.io/jenkins/ssh-agent:nanoserver-1809-jdk11 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: "1809" tags: - docker.io/jenkins/ssh-agent:nanoserver-1809-jdk11 nanoserver-1809_jdk17: image: docker.io/jenkins/ssh-agent:nanoserver-1809-jdk17 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: "1809" tags: - docker.io/jenkins/ssh-agent:nanoserver-1809-jdk17 nanoserver-1809_jdk21: image: docker.io/jenkins/ssh-agent:nanoserver-1809-jdk21 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: "1809" tags: - docker.io/jenkins/ssh-agent:nanoserver-1809-jdk21 nanoserver-ltsc2019_jdk11: image: docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk11 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk11 nanoserver-ltsc2019_jdk17: image: docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk17 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk17 nanoserver-ltsc2019_jdk21: image: docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk21 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk21 nanoserver-ltsc2022_jdk11: image: docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk11 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk11 nanoserver-ltsc2022_jdk17: image: docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk17 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk17 nanoserver-ltsc2022_jdk21: image: docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk21 build: context: . dockerfile: windows/nanoserver/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk21 windowsservercore-ltsc2019_jdk11: image: docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk11 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk11 windowsservercore-ltsc2019_jdk17: image: docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk17 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk17 windowsservercore-ltsc2019_jdk21: image: docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk21 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: "1809" WINDOWS_VERSION_TAG: ltsc2019 tags: - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk21 windowsservercore-ltsc2022_jdk11: image: docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk11 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-11 JAVA_VERSION: 11.0.23+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk11 windowsservercore-ltsc2022_jdk17: image: docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk17 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-17 JAVA_VERSION: 17.0.11+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk17 windowsservercore-ltsc2022_jdk21: image: docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk21 build: context: . dockerfile: windows/windowsservercore/Dockerfile args: JAVA_HOME: C:/openjdk-21 JAVA_VERSION: 21.0.3+9 TOOLS_WINDOWS_VERSION: ltsc2022 WINDOWS_VERSION_TAG: ltsc2022 tags: - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk21 ```

Current docker compose file content for comparison

Notice that almost all args currently have to be injected by build.ps1: https://github.com/jenkinsci/docker-ssh-agent/blob/bad064233ecebab858236cd141e39b06b5fdbf98/build-windows.yaml#L1-L39

Details

Avoid mistakes, oversights, inconsistencies or more work to do thanks to unique location of image definitions

Having only one place to store image definitions, build variables and helper functions would decrease the risk of mistakes or oversights, and reduce the work needed to change parameters common to both Linux and Windows, currently defined in two distinct files and a bit of data manipulations defined in build.ps1.

Ex: the work I've done for https://github.com/jenkinsci/docker/pull/1891 to remove Java 11 from Weekly release line would have been much more simpler.

Something like that:

```diff -variable "jdks_to_build" { - default = [11, 17, 21] +# Return the JDKs to build depending if the version matches LTS versionning pattern +function "jdkstobuild" { + param = [version] + # Return all JDKs only if the version matches LTS versionning pattern + result = equal(regex(version, ltsRegexPattern), "") ? [17, 21] : [11, 17, 21] } target "alpine" { matrix = { - jdk = jdks_to_build + jdk = jdkstobuild(VERSION) } target "debian" { matrix = { - jdk = jdks_to_build + jdk = jdkstobuild(VERSION) } target "nanoserver" { matrix = { - jdk = jdks_to_build + jdk = jdkstobuild(VERSION) } target "windowsservercore" { matrix = { - jdk = jdks_to_build + jdk = jdkstobuild(VERSION) } ```

Here is the code that would be required to remove Java 11 from Linux and Windows images here with the proposed enhancement:

variable "jdks_to_build" {
-   default = [11, 17, 21]
+   default = [17, 21]
}

Another place where it reduces complexity: the updatecli manifests, reducing the number of targets.

Implementing something like https://github.com/jenkinsci/docker-agent/pull/816 would result in a new Dockerfile argument and a docker bake function returning the correct WINDOWS_VERSION_DIGEST depending on the Windows version, no blocker from the enhancement.

• Demonstration: https://github.com/lemeurherve/docker-ssh-agent/pull/3/

Simplify build.ps1, which won't have to massage data for images definitions anymore

Currently, build.ps1 is used to massage the data for the image definitions and is in charge of the special case like this: https://github.com/jenkinsci/docker-ssh-agent/blob/bad064233ecebab858236cd141e39b06b5fdbf98/build.ps1#L39-L51

With this enhancement, build.ps1 would just be used for:

• (new) Optionally setting up the docker compose file generation from the docker bake definition without having to work on them
• Calling docker compose
• Optionally setting up the tests harness and calling Pester to run it

Apart from this generation, the rest of the process is unchanged, just an additional init step to generate the docker compose file, optional if it already exists.

When docker buildx will be able to build images on Windows, build.ps1 role will be:

• Calling docker buildx bake
• Optionally setting up the tests harness and calling Pester to run it

Contributors will be able to either use build.ps1 to (re)generate their docker compose file, or a command line that will be explained in the README to generate themselves the docker compose file. (See "miscellaneous" below)

With this proposed change, build.ps1 isn't mandatory for contributors anymore, they will be able to build and publish their own images by using docker bake and docker compose only.

Pave the way for when we'll be able to use docker buildx bake to build and publish images on Windows

BuildKit seems to be able-ish to build images on Windows:

• https://techcommunity.microsoft.com/t5/containers/experimental-windows-containers-support-for-buildkit-released-in/ba-p/4096116
• https://github.com/moby/buildkit/issues/616#issuecomment-2003071460

But docker buildx isn't yet.

Main issues to follow the progress:

• https://github.com/microsoft/Windows-Containers/issues/34

When it will be ready, we'll be able to remove the docker compose file generation and call and just use docker buildx bake to build and publish all images.

Miscellaneous

Example of the call to generate the docker compose file from a Windows machine:

$ docker buildx bake --progress=plain --file=docker-bake.hcl windows --print `
    | yq --prettyPrint '.target[] | del(.output) | {(. | key): {\"image\": .tags[0], \"build\": .}}' | yq '{\"services\": .}' `
    | Out-File -FilePath build-windows.yaml

• Use docker buildx bake to output image definitions from the "windows" bake target
• Convert with yq to the format expected by docker compose
• Store the result in the docker compose file

The only new requirements is docker buildx as docker, docker compose and yq are already required.

Corresponding change (missing the doc, incoming)

https://github.com/jenkinsci/docker-ssh-agent/commit/ed751ae2946c8898350e3b04985c824a017735b1

Working build locally and on ci.jenkins.io

https://ci.jenkins.io/job/Packaging/job/docker-ssh-agent/job/PR-415/88/

Upstream changes

No response

Are you interested in contributing this feature?

Yes:

• 415

[lemeurherve]

Build https://ci.jenkins.io/job/Packaging/job/docker-ssh-agent/job/master/444/ successful 🎉

I forgot to remove the docker compose target from updatecli manifests, opened https://github.com/jenkinsci/docker-ssh-agent/pull/430 that has been merged.

I'm triggering a new release to verify the publish process.

cc @jenkinsci/team-docker-packaging FTR.

[lemeurherve]

Release https://github.com/jenkinsci/docker-ssh-agent/releases/tag/5.41.0 completed on trusted.ci.jenkins.io with success 🥳

https://hub.docker.com/r/jenkins/ssh-agent/tags?page=&page_size=&ordering=&name=5.41.0