Migrate official build "Prepare Signed Artifacts" from Build to Pipeline Artifacts

[dagood]

The prepare signed artifacts stage downloads the full set of artifacts (nupkgs, etc) that all jobs contribute to, does some prep work on them such as signing, then uploads this large set back up to the build so that Arcade stages can publish them.

The reupload has been hitting some Azure rate limits and making official builds fail very frequently. We got a temporary rate limit increase but need to get to a better place. ICM 166161231, Msft email thread "AzDO Artifact Pipeline"

We don't yet know exactly how the rate limit works (API rate vs. size rate vs. ???), but we know that migrating to Pipeline Artifacts is the way to go.

This means:

• Stop using logging commands to upload these artifacts, because it's not supported with Pipeline Artifacts anymore: https://github.com/microsoft/azure-pipelines-tasks/issues/11028
  • This will cause the MSBuild code to be more tightly coupled with the yaml, but that isn't a huge concern for me with this specific stage/job.
• Upload all the artifacts in one go, probably with PublishPipelineArtifact@1
• Still use logging commands to upload "tricky" files like the asset manifests. This is Arcade infrastructure that we should handle uniformly across the repos IMO. We could try moving the manifests to a staging directory, assembling a manifest of things to upload that we handle ourselves in yaml, or something like that, but let's leave that to another time. These are small and few.

Other places have more significant reasons to use logging commands, so they'll be harder to switch over. The prepare signed artifacts stage is low-hanging fruit. I think it'd take me a handful of days.

/cc @dotnet/runtime-infrastructure

[jaredpar]

I thought #8582 was tracking a different AzDO bug. Essentially there are two AzDO publish issues at the moment:

1. Throttling where the recommendation is to move towards Pipeline Artifacts. There is an email + ICM about this but no issue I was aware of on core-eng
2. https://github.com/dotnet/core-eng/issues/8582 which tracks a bug in AzDO or Azure SDK which was a separate issue.

Do you believe they're the same?

[dagood]

I got it mixed up--edited to replace that bad link with a link to the ICM ticket from https://github.com/dotnet/runtime/issues/702.

[jaredpar]

Thanks! Mostly wanted to make sure I was tracking the proper set of issues.

[safern]

Arcade also does a lot of uploading/downloading, should we open an issue in arcade to switch to PipelineArtifacts?

[dagood]

At this point I don't think Arcade contributes significantly to the rate limit issue dotnet/runtime has been hitting (although I have basically no data on this yet), but yes, AFAIK everyone should migrate. Filed https://github.com/dotnet/arcade/issues/4628.

[dagood]

Heads up that I'm not treating this as a critical issue and I'm working on a .NET 5 feature work blocker first.

Mitigations:

• We have a rate limit increase in effect until 2020-1-31 00:00z. We can probably extend it.
• Arcade folks don't think they have a clear path forward yet, and we don't even know for sure yet that improving Prepare Signed Artifacts will be enough.
• Someone else can pick it up. This bit of the build isn't too complex, basically rip out the logging commands here and add a step to prepare-signed-artifacts.yml: https://github.com/dotnet/runtime/blob/7b92525c75e245bd8badb27359ac41ef6b6ff911/src/installer/publish/prepare-artifacts.proj#L44-L51

[dagood]

Glancing at this again, I realize that I'm pretty sure PreparedArtifacts can be cut entirely... it was being uploaded to support some custom publish infra that isn't present anymore.

The PushToAzureDevOpsArtifacts Arcade SDK task does essentially the same logging command uploads, but puts the bits in the right places (BlobArtifacts and PackageArtifacts) for the Arcade stages to pick up.

So I think we should cut PreparedArtifacts to reduce the redundant load we're putting on the system. Beyond that, though, we're stuck until Arcade and AzDO figure something out.

[NikolaMilosavljevic]

[Triage] @jkoritzinsky do you know if this is tracked as part of leg consolidation and optimization?

[jkoritzinsky]

I'm not sure. I'm not familiar with this issue. @safern or @jaredpar would know better.

[safern]

I don't thin kit is part of the leg consolidation, however we should do this after the leg consolidation is done.

[ViktorHofer]

We might not need build or pipeline artifacts anymore at all after 1) the legs are consolidated and 2) the "Prepare Signed Artifacts" leg is eliminated: https://github.com/dotnet/runtime/issues/44842.

[dagood]

At this point I don't think Arcade contributes significantly to the rate limit issue dotnet/runtime has been hitting (although I have basically no data on this yet), but yes, AFAIK everyone should migrate. Filed https://github.com/dotnet/arcade/issues/4628.

FYI core-eng closed https://github.com/dotnet/arcade/issues/4628 due to the move to GitHub Actions. (I'm not familiar with this.)