dotnet / docker-tools

This is a repo to house some common tools for our various docker repos.
MIT License
103 stars 42 forks source link

Define isolated Auto Builder subscriptions for buildtools-prereqs repo #1210

Open mthalman opened 4 months ago

mthalman commented 4 months ago

The Auto Builder subscription for the dotnet/dotnet-buildtools-prereqs-docker is currently configured such that it handles updates for all Dockerfiles in the repo. This can cause a bottleneck when issues occur that may prevent images for other unaffected distros from getting published. For example, let's say that both Ubuntu and Mariner have base image updates that Auto Builder kicks off a build for. The build of the Ubuntu Dockerfiles happen to fail to some misconfiguration. But the Mariner Dockerfiles are unaffected and all pass their builds. This prevents Mariner from being updated until the Ubuntu issue is fixed.

The repo is already configured with OS-specific sub-manifests (example). So a way to help mitigate issues like this is to update the subscription to target those sub-manifests instead of the main manifest. Then the pipeline being targeted in the subscription can also be updated to the OS-specific pipeline. Having distinct pipelines is an important aspect of this since the Auto Builder logic bases its retry count on a per-pipeline basis. So if the pipeline fails to run 3 times in a row, it stops trying anymore. This would allow each OS to be handled independently wrt Auto Builder.

mthalman commented 4 months ago

I ran a quick experiment to see if things would just work with a reconfigured subscription file. It doesn't just work.

Here's a sample entry from the config:

  {
    "manifest": {
      "owner": "dotnet",
      "repo": "dotnet-buildtools-prereqs-docker",
      "branch": "main",
      "path": "src/centos/manifest.json",
      "variables": {
        "FloatingTagSuffix": "",
        "UniqueId": ""
      }
    },
    "imageInfo": {
      "owner": "dotnet",
      "repo": "versions",
      "branch": "main",
      "path": "build-info/docker/image-info.dotnet-dotnet-buildtools-prereqs-docker-main.json"
    },
    "pipelineTrigger": {
      "id": 437,
      "pathVariable": "imageBuilder.pathArgs"
    }
  }

The key here is that the manifest path is pointing to the manifest specific to CentOS.

It caused this exception:

Unhandled exception: System.IO.FileNotFoundException: Path specified in manifest file does not exist.
File name: 'src/centos/7/Dockerfile'
   at Microsoft.DotNet.ImageBuilder.ViewModel.ModelExtensions.ValidateFileReference(String path, String manifestDirectory) in /image-builder/src/ViewModel/ModelExtensions.cs:line 119
   at Microsoft.DotNet.ImageBuilder.ViewModel.ModelExtensions.ValidatePlatform(Platform platform, String manifestDirectory) in /image-builder/src/ViewModel/ModelExtensions.cs:line 163
   at Microsoft.DotNet.ImageBuilder.ViewModel.ModelExtensions.ValidateImage(Image image, String manifestDirectory) in /image-builder/src/ViewModel/ModelExtensions.cs:line 155
   at Microsoft.DotNet.ImageBuilder.ViewModel.ModelExtensions.ValidateRepo(Repo repo, String manifestDirectory) in /image-builder/src/ViewModel/ModelExtensions.cs:line 147
   at Microsoft.DotNet.ImageBuilder.ViewModel.ModelExtensions.Validate(Manifest manifest, String manifestDirectory) in /image-builder/src/ViewModel/ModelExtensions.cs:line 63
   at Microsoft.DotNet.ImageBuilder.ViewModel.ManifestInfo.Create(String manifestPath, ManifestFilter manifestFilter, IManifestOptionsInfo options) in /image-builder/src/ViewModel/ManifestInfo.cs:line 63
   at Microsoft.DotNet.ImageBuilder.ViewModel.ManifestInfo.Load(IManifestOptionsInfo options) in /image-builder/src/ViewModel/ManifestInfo.cs:line 46
   at Microsoft.DotNet.ImageBuilder.SubscriptionHelper.GetSubscriptionManifest(Subscription subscription, ManifestFilterOptions filterOptions, IGitService gitService, Action`1 configureOptions) in /image-builder/src/SubscriptionHelper.cs:line 78
   at Microsoft.DotNet.ImageBuilder.SubscriptionHelper.GetSubscriptionManifests(String subscriptionsPath, ManifestFilterOptions filterOptions, IGitService gitService, Action`1 configureOptions) in /image-builder/src/SubscriptionHelper.cs:line 34
   at Microsoft.DotNet.ImageBuilder.Commands.CopyBaseImagesCommand.ExecuteAsync() in /image-builder/src/Commands/CopyBaseImagesCommand.cs:line 62

I'm guessing what's happening is that it's trying to load the Dockerfile relative to the manifest file. But the Dockerfile paths in the manifest are relative to the repo root.