All publishing broken until Maestro rollout completes

mmitche commented 1 month ago

The publish using darc steps are failing right now because the darc client that is in use by builds (specified by prod) has a DI bug. This DI bug was fixed by #3717, but rolling out got blocked by a couple issues:

Maestro couldn't use the prod darc in the "publish using darc" stage, and so the prod rollout failed. This was worked around with #3723
Firewall rules broke the ability to deploy database updates (see past attempts of https://dev.azure.com/dnceng/internal/_build/results?buildId=2493203&view=results).

The original issue here is that prod maestro was able to be rolled out without a working darc in maestro-int. This was because this old prod darc was used to publish the new arcade-services build. As soon as the rollout completed, all publishing broke. This meant that there was no possibility of a rollback without at least some additional fix. So how do we avoid this in the future?

We probably at least need a scenario test that checks that publishing works with the maestro-int version of darc. The prod version of darc should still be used in the rollout pipeline, so that a bad maestro-int doesn't block future rollouts.

mmitche commented 1 month ago

@pavel-purma @premun @dkurepa

dkurepa commented 1 month ago

yeah, testing add-build-to-channel definitely sounds like a must

mmitche commented 1 month ago

Should be resolved now.

dkurepa commented 1 month ago

adding a test for the just built darc in this PR https://github.com/dotnet/arcade-services/pull/3725

dotnet / arcade-services

All publishing broken until Maestro rollout completes #3724