Prevent spurious breakage when tagging happens

dlang / ci

dlang CI testing pipelines

https://buildkite.com/dlang

Boost Software License 1.0

13 stars 29 forks source link

Prevent spurious breakage when tagging happens #331

Open Geod24 opened 6 years ago

Geod24 commented 6 years ago

At the moment, we frequently hit the following issue:

Buildkite builds a project
Some change is made in the project (e.g. in .travis.yml)
Nothing happens for a while
A new release is tagged
The Buildkite build is broken, stalling all auto-merge in DMD/druntime/phobos (although not blocking direct merge)
The person around has to find a solution, which is either fixing buildkite, disabling the test, or fixing the lib + getting a patch release

As the number of project grows, so does this issue.

We could set as a requirement that a project being built via Buildkite also uses Buildkite itself as one of its CI, but that will obviously trigger more issues (especially since the logs aren't public). Another idea would be to have a fallback to the previous version for the next X hours after a new release is created, but have dlang-bot create an issue in this repo tagging the maintainer and us.

Any thought @wilzbach ?

MartinNowak commented 5 years ago

Projects that regularly regularly release broken test-suites should be removed from the project-tester. If OTOH projects regularly get broken by dmd/druntime/phobos changes, then those failures are legitimate and should rarely be worked around.

Geod24 commented 5 years ago

Projects that regularly regularly release broken test-suites should be removed from the project-tester.

The thing is, it's not broken for them. They do a legitimate change to their project test suite. I've never seen a project release a broken release, it was always some change in the settings (e.g. installing a new dependency in their .travis.yml) that broke stuff.

MartinNowak commented 5 years ago

So you're talking about that angle. Indeed new dependencies might break builds, do you have a good example? I think the simplest way to solve this for now is by communication, there are not so many projects with external dependencies (as those hardly integrate with dub anyhow).

Also we've pre-installed quite a list of dependencies and I just linked to that in the README (https://github.com/dlang/ci/blob/master/README.md#pre-installed-dependencies-on-buildkite-agents).

PetarKirov commented 5 years ago

Why not simply require that all projects are build in containers? That way won't have any dependency management to do on our side, except for the container runtime (e.g. Docker, Windows Containers, etc.)

Geod24 commented 5 years ago

@MartinNowak : The only breakages I've seen so far have been just this. A dependency to the build gets added, but we don't have it. E.g. we had meson builds being added in 2 cases ( https://github.com/dlang/ci/pull/307#issuecomment-420479394 ), and now a rewritten test-suite.