Investigate alternatives to publishing artifacts via logging commands in the Arcade SDK

dotnet / dnceng

.NET Engineering Services

MIT License

25 stars 18 forks source link

Investigate alternatives to publishing artifacts via logging commands in the Arcade SDK #2371

Open riarenas opened 7 months ago

riarenas commented 7 months ago

[ ] This issue is blocking
[ ] This issue is causing unreasonable pain

Currently, the publishing infrastructure in the Arcade SDK uses the Artifact upload logging commands to publish intermediate artifacts to the PackageArtifacts and BlobArtifacts containers.

https://github.com/search?q=repo%3Adotnet%2Farcade+artifact.upload&type=code

Artifacts uploaded via this mechanism skip the scans that the 1ES pipeline templates enforce on each artifact upload.

We should look into alternatives that allow the 1ES templates to scan the artifacts the Arcade SDK produces

dougbu commented 7 months ago

during our "deep dive" meeting, the 1ES team seemed surprised logging commands existed and were unsure whether they could intercept them (somehow). do we know if they're working on something here❔

today's ##vso[artifact.upload] commands enable easy flattening of a folder tree. we use this feature in PushToBuildStorage for the AssetManifests, BlobArtifacts, and PackageArtifacts artefacts. we explicitly do not flatten the tree when creating the PdbArtifacts artifact. need a replacement approach which supports both choices.

separately, this logging command creates or updates a build artefact. I seem to remember it also does retries under the covers and avoids scaling problems. we'll need to stress test a replacement to ensure we don't regress our reliability.

missymessa commented 7 months ago

cc @tkapin

missymessa commented 6 months ago

Moving this to dotnet/dnceng so we can put it in an epic.

dougbu commented 6 months ago

Move makes even more sense now that I've noticed some of our code e.g. in dotnet-helix-machines (see VstsBuildContextLogger in particular) also uses ##vso[artifact.upload ...] logging commands.

missymessa commented 6 months ago

[ ] Start a conversation with 1ES
- [ ] If 1ES doesn't have the bandwidth here, start a conversation with AzDO
[ ] Come up with a design (in case we need to implement a solution in an emergency)
[ ] determine priority after the above items are complete

missymessa commented 6 months ago

Also see chat from the "Publishing Artifacts Alternatives" meeting.