search

dotnet / docker-tools

This is a repo to house some common tools for our various docker repos.
MIT License
122 stars 46 forks source link

Migrate to 1ES pipeline templates #1215

Closed mthalman closed 6 months ago

mthalman commented 6 months ago

Updates the dotnet-buildtools-image-builder-official.yml pipeline to the 1ES templates.

In order to share the templates between public and internal pipelines, there needed to be a common way to publish artifacts. This is done by encapsulating that logic into a new template: /eng/common/templates/steps/publish-artifact.yml. This uses conditions to use the appropriate publish task based on whether its internal or public.

In order to get this to work on Linux Arm64 machines, it was necessary to add hostArchitecture: Arm64 to the pool definition. Otherwise, it was getting an error when installing CredScan since CredScan doesn't support Arm64 yet. Setting the hostArchitecture field this way prevents an attempt to install CredScan.

The new 1ES images we're using have multiple versions of ManifestGeneratorTask installed. This broke the logic in the Generate SBOMs step of build-images.yml because it doesn't expect there to be multiple version directories there. I updated that to account for this.

A change was also needed in the Consolidate SBOMs to Single Directory step of post-build.yml. Because the SBOM artifacts that are uploaded for each build job now contain an SBOM for the artifact itself (not the Docker image). This causes directory collisions when consolidating them in this step. So it uses -Force to allow overwriting in the destination as this directory that's colliding isn't even used anyway. We're just taking the SBOM for the Docker image out of the extracted location.   /cc @dagood

dotnet-issue-labeler[bot] commented 6 months ago

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

qmuntal commented 6 months ago

Great work! I've tested the updated templates and it mostly works. There is one recurrent error in the 1ES.PublishPipelineArtifact task, though. It doesn't accept file paths, only directories, else it can't generate a SBoM manifest for the artifact (example).

Arcade also hit this error and fixed it by putting the artifacts to upload in its own folder: https://github.com/dotnet/arcade/pull/14582.

mthalman commented 6 months ago

Great work! I've tested the updated templates and it mostly works. There is one recurrent error in the 1ES.PublishPipelineArtifact task, though. It doesn't accept file paths, only directories, else it can't generate a SBoM manifest for the artifact (example.

Arcade also hit this error and fixed it by putting the artifacts to upload in its own folder: dotnet/arcade#14582.

Yes, I forgot to mention this in my original post. I'm going to log an issue for this once this is merged. This will be follow up work. That error isn't blocking things and we've got two other people on our team that are waiting on these changes so we can work in parallel on other pipelines.