[infra] Many builds are stuck in queue despite having idle machines in the pool

[harryterkelsen]

Type of Request

bug

Infrastructure Environment

Not completely sure, but seems like an issue with the Cocoon scheduler

What is happening?

Many builds are stuck in queue on commit 96d9cd1195917f66e7341f34eeb146d1a93a3767

Even though it looks like there are many idle machines in the pool:

Steps to reproduce

See the build is stuck https://flutter-dashboard.appspot.com/#/build

Expected results

Builds should be scheduled and tests run when there are idle machines

[zanderso]

Previous similar issue: https://github.com/flutter/flutter/issues/145939

[zanderso]

Over email @godofredoc has suggested rolling back today's dashboard deployment.

[christopherfujino]

Over email @godofredoc has suggested rolling back today's dashboard deployment.

I rolled back the cocoon version to one from April 23. Still waiting to verify if the new builds succeed.

[zanderso]

The tree is open, but due to some inconsistency in the task results database(s), the cocoon scheduler is batching jobs without backfilling for some tests. Since the tree is open, I am dropping this down to P1.

[christopherfujino]

Related to https://github.com/flutter/flutter/issues/142951

[stuartmorgan]

It looks like several builds that were started yesterday afternoon got into the inconsistent DB state again, so this isn't fully resolved. I'm not sure yet if there were two causes and the rollback only fixed one, or if the PRs that were rolled back were a red herring.