search

dotnet / arcade-services

Arcade Engineering Services
MIT License
61 stars 74 forks source link

[darc/maestro++] dependency flow: how to deprecating a package ID? #2569

Open natemcmaster opened 5 years ago

natemcmaster commented 5 years ago

This is a fairly common scenario for dev work, and I don't see a way to do this with darc yet. With my understanding of the current design, the Maestro bot will leave behind stale references to deprecated packages when either of these scenarios occur. In ProdCon v1, we had a protection against this -- a strict check on incoming version variables -- which helped us know when packages were dropped or renamed. Unless I've missed something, I don't see a replacement for this in darc/maestro.

Scenario A - renaming a package on a channel

Team A was producing previews of System.Banana and pushing to channel ".NET Core Dev", but decides they would rather ship this as System.Strawberry. Consumers of need to update to the new package ID.

Scenario B - deleting a package from a channel

Team A was producing previews of System.Titanic, but due to reasons, has decided this package should not ship (pun intended). Consumers should stop using this package as it will not have a production release.

Questions

cc @markwilkie @jcagme

markwilkie commented 5 years ago

There was some talk of cleanup before, but I don't remember where it was left off. @mmitche

mmitche commented 5 years ago

In prodcon v1 we were very much concerned about this because the source did not describe what packages were actually inputs to the build. In other words, if the package stopped appearing in the 'channel' (upstream build), then you could effectively downgrade the package without knowing. Prodcon v2 doesn't suffer from this as much.

It's entirely valid that packages would stop being produced in a channel, like if we service package X in 3.0.5 but not in 3.0.6. so how do we differentiate these cases? I do like the idea of deprecation/removal though. Basically saying that "components should not take a dependency on this package any longer". That would cover both your scenarios above.

In my mind, this gets implemented as:

This means that other channels (e.g. a servicing channel) work just fine still, as maybe System.Strawberry is still there and being produced.

natemcmaster commented 5 years ago

Ok, here is a concrete example. I'm not seeing updates to System.Threading.Tasks.Extensions on the '.NET Core 3 Dev' channel to aspnet/Extensions. I've just noticed this. What should I do? Email the corefx team to see if they dropped this package? Investigate darc to see if this is a Maestro bug? Remove the dependency and pin to an LKG version?

mmitche commented 5 years ago

I would probably email the corefx team in this case. It's not being produced.

mmitche commented 5 years ago

I believe it's part of Microsoft.NETCore.App

natemcmaster commented 5 years ago

Thanks. How did you figure out that it's not being produced anymore? My main point was that nothing in the system makes it easy to figure this out.

mmitche commented 5 years ago

I used the swagger API (https://maestro-prod.westus2.cloudapp.azure.com/swagger/ui/index.html#/Builds/Builds_GetLatest) to look up the latest corefx build.

mmitche commented 5 years ago

Though, I will say that nothing in corefx is really easy to figure out. It's one reason why we have so many issues shipping bits incorrectly from there.

natemcmaster commented 5 years ago

For this specific case, I've opened https://github.com/dotnet/corefx/issues/34473

jcagme commented 5 years ago

I have some thoughts about this that would involve maintaining a baseline per repo and then getting a delta of what is included in the latest pushes. When somethings is detected to be missing we notify the repo owners and once deprecation/rename is confirmed then we update the baseline. Let's talk more about this in person when we have a chance