Improve Continuous Integration for repositories (build on schedule)

[dnsl48]

Description

Some repositories aren't getting commits for long time. For example, some branches of recipe-cms may not get committed to for months. Even if we have a red build, it stays green on the dashboard until we trigger a new build. That's not cool and we should consider running tests on schedule (weekly?).

[ScopeyNZ]

You can schedule builds on travis so this should be an easy addition.

[dnsl48]

Yes. We just need to document this decision, schedule and the repos we're doing it for.

[ScopeyNZ]

I'm inclined to say "just do it" for repos that don't get runs often. Just be aware that you should probably set aside some time on a Sunday morning (or something) to schedule them as they'll run every from the time you first schedule.

[ScopeyNZ]

I don't really think we need to RFC this but I'll put the relevant tags on anyway.

[robbieaverill]

I agree, do it. Set it for overnight NZ time on a weekend

[dnsl48]

I'm thinking about early Saturday morning. I foresee that lots of modules running simultaneously will be exhausting our Travis capacity, but picking separate times for them would be too much maintenance overhead.

[dnsl48]

for repos that don't get runs often

Monitoring what gets run often is also going to be maintenance overhead. I reckon we should do this for everything without any exceptions (otherwise we would have to maintain the exceptions list somewhere).

[dnsl48]

There's some potential overlap with continuous merge-ups RFC https://github.com/silverstripe/silverstripe-framework/issues/9083

[dnsl48]

FYI: I've set up weekly auto build last Saturday morning for the open source recipe repositories (except CWP ones).

[ScopeyNZ]

We can close this imo. We can raise an issue on a CWP repo if we want to do this for CWP.

[dnsl48]

We can close this imo

I only done it for some recipes though. I'm not sure that's all we need to achieve. Ideally this should be done for everything that has external dependencies and also we should make it a part of our CI practice in general for all new repositories as well.

[ScopeyNZ]

We can close this imo

I only done it for some recipes though. I'm not sure that's all we need to achieve. Ideally this should be done for everything that has external dependencies and also we should make it a part of our CI practice in general for all new repositories as well.

Right, sorry, I missed recipe in your comment 😅

[brynwhyman]

This has the rfc/draft label. Can a core committer please tag the group for consensus, or do we agree this doesn't need that label? We're keen to get this done now.

[robbieaverill]

@silverstripe/core-team

[robbieaverill]

I’d suggest building one branch rather than all of them. Not sure whether the current major version branch or latest minor would be better. The former requires less maintenance but the latter would be more useful

[brynwhyman]

Yeah, if we're going to choose one it seems like a regular build on the latest minor is going to be more useful. If the effort involved in regularly maintaining this is a worry, it's worth noting that we want to be working on the following in the next couple of weeks (which might help): https://github.com/silverstripe/silverstripe-framework/issues/9174

[chillu]

+1 on building one branch, there's a limit to how much we can maintain. The current major version seems to be the best bang for our buck there, since it'll also highlight which modules aren't "release ready" (for a new minor release).

Can a core committer please tag the group for consensus, or do we agree this doesn't need that label?

@brynwhyman I think this group has been changed to "public" a while ago, so anyone can tag the core team now.

Is there a way to monitor how this affects queue depth throughout the times when we're relying on it most? So NZ working hours. I didn't see this as an org-wide metric in Travis anywhere, and doing it via API is probably too much effort.

[dnsl48]

The unfortunate bit here is that through TravisCI UI we don't have any control over when to run the builds. The current behaviour (which is not guaranteed by Travis) is to run it at the same time when it's been updated manually through the UI. This means if we want to run it on Sunday mornings, we have to update it manually in the UI on Sunday morning, and then we don't have any guarantees that it's going to keep running those builds on Sundays and not gonna change it to another day or time.

[robbieaverill]

Most builds other than installer, kitchen sink, and framework don't take that long to run, so if they were randomly spread out it probably won't affect you too much

[chillu]

I think running the recipes weekly on a random schedule is perfectly fine. Looking at recipe-cms, that already runs tests for all it's dependencies, on their -dev branches.

Use case: An unreleased change on a dependency like framework in the current major release branch broke behaviour in a module relying on it. The module itself hasn't had commits in a while. We can catch this change before it finds its way into a stable release through scheduled builds. We don't have an explicit process for re-running all builds before a release, so this might get missed otherwise.

I think this use case is covered without individual scheduled module builds (on top of the recipe builds). Unless anyone feels strongly about adding scheduled module builds, I'd suggest we close this issue.

[dnsl48]

Yes, agree, running recipes should be enough. If we get any more automation around setting auto-builds for a hundred of repositories, we can revisit this topic. Otherwise, I think it's good enough already and not worth the effort of manually configuring all of those.