dotnet / source-build

A repository to track efforts to produce a source tarball of the .NET Core SDK and all its components
MIT License
266 stars 132 forks source link

Publish Artifacts CI build step does not support retries #4537

Closed MichaelSimons closed 2 months ago

MichaelSimons commented 3 months ago

The Publish Artifacts CI build step will fail if rerun when a previous attempt has uploaded any artifacts. This is because the publish build step is set to run on success or failure and that it does not support overwriting artifacts. When the occurs, the failure will will look similar to the following:

##[error]Artifact CentOSStream9_Online_PreviousSourceBuiltSdk_x64_Artifacts already exists for build 2502184.

There are a couple options to solve this but they have some cons:

  1. Only publish on success. This would work for normal build failures but if the 1ES steps fail which are run after publish, re-run will still encounter this issue. Furthermore, in some instance it is desirable to inspect the artifacts that were produced during a failed build. This change would prevent that.
  2. Modify the Publish Artifacts step to include the Job Attempt number. This would mean each run would have it's artifacts uploaded to a distinct artifact name. The downside to this, is there are some dependent jobs (e.g. the source-build sdk content tests) which download the artifacts. They would need to account for the possibility of multiple artifacts existing.
ellahathaway commented 2 months ago

A very similar issue was occurring in docker: https://github.com/dotnet/docker-tools/issues/1323

Here is a summary of the options & their respective challenges:

Frankly, none of the options are great. In an ideal world, we could delete the old artifacts and upload new ones. But since that's not possible, I think that the best way forward is to condition the publishing of the artifacts on the success of previous step. It poses the least risk and complications, and we'd still have the logs and binlogs from previous attempts so if there's a failure on a previous attempt, we'd still be able to investigate. This is also the solution that was decided upon in https://github.com/dotnet/docker-tools/issues/1323.

MichaelSimons commented 2 months ago

I agree @ellahathaway with your assessment.