EOL annotation race condition

[mthalman]

When EOL annotation support is added to the dotnet-buildtools-prereqs-docker repo, it can result in a race condition when determining which images should be annotated, potentially resulting in supported images being EOL-annotated.

This repo is unique in this aspect because of its use of multiple pipelines that all write to the same image info file. When there are multiple pipelines running concurrently, there can be the following race condition scenario:

1. Debian pipeline starts
2. Fedora pipeline starts
3. Debian pipeline pushes new images to ACR (but hasn't yet updated image info file)
4. Fedora pipeline pushes new images to ACR
5. Fedora pipeline updates image info file
6. Fedora pipeline queries to find images that need to be annotated. It finds the Debian images and sees they are not in the image info file.
7. Fedora pipeline sets EOL annotations, which includes newly published Debian images

Possible solutions:

• Have detection logic in the pipeline to see whether this scenario occurred and throw an error. This doesn't prevent the scenario but simply informs that this happened, allowing a team member to rebuild the effected images. In the example above, the Debian pipeline would check whether any of the images it published had been marked as EOL and it would run that check after updating the image info file.
• Prevent multiple pipelines from running concurrently. This may mean going back to a single pipeline.
• Define separate image info files for each pipeline. This would mean having separate container repos for each of them.
• Define some wait behavior in the pipeline so that it won't proceed with querying for images to be annotated if there have been recent pushes (within X minutes) to the registry outside the context of the images currently being published by that pipeline.
• Use pipeline tags as semaphores to indicate the pipeline status so that multiple concurrent runs can "communicate" with each other. Using the example above, Debian pipeline would add tags when it has started to push images and after it had updated the image info. For the Fedora pipeline, it would wait to proceed with querying for images to be annotated until after any other concurrently running pipeline had those two tags set.

[lbussell]

[Triage] We should investigate if option 2 (going back to a single pipeline) is feasible without wasting a huge amount of compute (via any Azure DevOps features, etc.)

Another way to go back to a single build pipeline could be to compute which images will be cached/built ahead of build matrix generation, so that we avoid spinning up too much compute for each individual build leg.

[mthalman]

Another way to go back to a single build pipeline could be to compute which images will be cached/built ahead of build matrix generation, so that we avoid spinning up too much compute for each individual build leg.

I'm going with this approach.