Auto Cancellation : Get faster results by only testing your most recent build

joshk commented 7 years ago

It's not uncommon to have your waiting builds queue grow, especially if your team is pushing new code to GitHub every few minutes.

With Auto Cancellation, you can enable Travis to cancel waiting and outdated builds when new builds flow in from GitHub.

This setting can be enabled for Branch and Pull Request builds separately.

Builds will only be canceled if they are waiting to run, allowing for any running jobs to finish.

You can find this new setting on the Repository settings page.

This feature was released for beta testing soon on the 22nd of March. You can find the blog post announcement here: https://blog.travis-ci.com/2017-03-22-introducing-auto-cancellation

Please Note: We are looking at adding the ability to also cancel running jobs. If this is of interest to you and your team, please add the 🎉 emoji to this issue 😄

svenfuchs commented 7 years ago

This is now live, and documented: https://docs.travis-ci.com/user/customizing-the-build/#Building-only-the-latest-commit

Blog post: https://blog.travis-ci.com/2017-03-22-introducing-auto-cancellation

PeterDenton commented 7 years ago

If there are pushes on different branches will there be cancellations? Example: I push up my dev branch with a new commit. I then merge it into my master branch and push that up. Meanwhile, the dev branch is queued and a new push arrives on the master branch. Does the dev branch still run? The description here and on the broadcast are a bit unclear on this point.

In any case, nice update, thanks.

crunis commented 7 years ago

Is there a way to activate this via API? we want it in all our builds!

@PeterDenton if that was the case, I would consider it a bug. The announced feature says PER BRANCH or PER PR, not PER WHOLE REPO.

joshk commented 7 years ago

Hi @PeterDenton and @crunis

@PeterDenton, @crunis is correct that this is applied per branch. Supporting the use case you described isn't super simple, but can be something for us to consider for the future.

@crunis yes, we do have an API. You can find docs at https://developer.travis-ci.org, specifically https://developer.travis-ci.org/resource/user_setting#User%20setting. Although I notice that the docs need some updating (we are releasing these docs next week, along with our new V3 API)

crunis commented 7 years ago

@joshk managed to use the API, thanks !

The "discoveries" I made that I would like to see in the documentation:

GET /repo/{repository.id)/settings returns all the available attributes
The 2 new attributes are auto_cancel_pushes and auto_cancel_pull_requests
One can update their values with something like : curl -X PATCH -H "Content-Type: application/json" -H "Travis-API-Version: 3" -H "Authorization: token xxxxxxx" -d '{ "user_setting.value": true }' "https://api.travis-ci.com/repo/{repository.id}/setting/auto_cancel_pull_requests"

repository.slug can be used instead of the repository.id, but then we have to use %2F as the user / repo_name separator

joshk commented 7 years ago

Thanks for the feedback @crunis

I've looped in @renee-travisci who is part of our awesome engineering team, and leads a lot of the development on the API.

Glad you got it all working!

jleclanche commented 7 years ago

@joshk Would also love this in all my builds. Is there a plan to turn it on by default? It's a lot of work to change every single repo setting, and then keep changing it for any new repo we add.

joshk commented 7 years ago

@jleclanche Once this is out of Beta this is definitely something I would be interested in. Thanks for the suggestion!

lbergelson commented 7 years ago

👍 We've been wanting a feature like this for a while.

LiNk-NY commented 7 years ago

@joshk, I would also suggest that this be the default setting. It conserves precious resources. I'm grateful for the Travis-CI project and team! :smile:

joshk commented 7 years ago

Hey @LiNk-NY, yep, that was my thinking too! 😃

grooverdan commented 7 years ago

Nice.

Builds will only be canceled if they are waiting to run, allowing for any running jobs to finish.

What isn't cancelled is waiting jobs that are part of a matrix build where one job has started. I.e. if any job has started in a matrix build the Auto Cancellation will wait for the entire matrix to finish.

I think the outstanding jobs of a build matrix should be cancelled but still allow the started jobs to complete.

megies commented 7 years ago

@joshk, I would also suggest that this be the default setting. It conserves precious resources. I'm grateful for the Travis-CI project and team! 😄

+1 For "default setting" and/or possibility to set it in the yaml. Thanks for the feature in any case @joshk!

krymen commented 7 years ago

Great work Travis team! I agree with @LiNk-NY and @megies -- it should be the default setting.

Fryguy commented 7 years ago

Very happy to see this...can't wait to delete our bot code

joshk commented 7 years ago

Hi @grooverdan, thanks for the feedback!

The current iteration of this feature allows for builds to finish if they have started. There is a possibility, especially if you have a busy team, that a build might never finish because your build (which might run for a long time) always gets cancelled by a new build.

The second iteration of this feature will allow for running builds to be automatically cancelled when a new build comes in.

@megies Thanks for the feedback. We are unlikely to add this to the .travis.yml we want to keep it a bit more focused on build configuration vs account settings.

drdanz commented 7 years ago

Very useful feature, thanks!

I'd like to have a better control on the branches where this is enabled, in particular I'd like to be able enable it for all branches with the exception of the main branches, for which I want to test all the commits.

Terry-Weymouth commented 7 years ago

You guys rock. Must have seem me and other frantically reaching for that cancel button. Maybe setting on by default on, with override in settings and/or yaml file? That would be great!

grooverdan commented 7 years ago

@joshk I'm not sure you have the same concept in mind as me. The concept i'm trying to convey (with job overloaded too many ways so I'm hoping this is consistent with current terminology) is: when a build starts it starts up to 5(4?) jobs. There may be more in the build same build that are there and haven't started. Auto Cancellation should cancel any jobs that are part of the same build that haven't started and leave the remaining jobs to finish.

ljharb commented 7 years ago

This feature is awesome!

I already submitted this feedback via another channel, but:

I can't safely use "auto cancel branch builds" unless I can enumerate the branches on which I never want this to happen - in other words, master, gh-pages, etc. I only want autocancellation on feature branches.

robdvr commented 7 years ago

Thank you for rolling this out! Great work!

nclopezo commented 7 years ago

Great idea, thank you!

Dead2 commented 7 years ago

I would like for it to prioritize the latest commit, and only cancel the older ones if the current one passes. If the latest one fails, work backwards until it finds a commit that succeeds, and only cancel the ones older than that one.

This retains more of the usefulness of building multiple commits, while still avoiding doing more builds than are needed.

joshk commented 7 years ago

@ljharb @drdanz thanks for the feedback! I'll take note of this and think about it more as I can see the benefit of such a feature.

@Dead2 that is a great idea, almost like a git-bisect for CI. This might be the gold standard of this feature, so I'm not sure if or when this might be possible. Thank you for the great idea!

@grooverdan I have to admit I'm still confused. A Build has one or more Jobs. The way Auto Cancellation works right now is that it will only cancel entire Builds which have not started (no Jobs have started). The next iteration will allow for Auto Cancellation to cancel Builds which have started (one or more Jobs which have started).

ljharb commented 7 years ago

@joshk i think what they're asking for (which would be nice) is for autocancellation of pending jobs, not pending builds.

joshk commented 7 years ago

@ljharb I think it would be handy to understand the use case of this feature, along with the flow of events and how it would work, as I don't really understand the idea or benefit.

What also plays out in my mind is how to make this option clear and understandable, and if this addition would be used by a small number of users.

ljharb commented 7 years ago

For me, the use case would be avoiding the slow part (waiting for jobs to start) while maximizing the number of jobs that actually complete (for maximal coverage).

I agree it would be very difficult to communicate the functionality clearly.

t1 commented 7 years ago

cool feature! maybe a global setting for the default would be nice.

metaskills commented 7 years ago

Again, another up vote for making this the default. From an open source perspective, it would seem beneficial to both end users as well as the TravsCI team to lower resource on y'alls end to support Open Source. Till then, I've spent some time going into each personal and organizational settings and turning this on. It is a default or me already :) CHEERS!

amc6 commented 7 years ago

It would be nice if you could customize this behavior on a per branch basis. In particular it would be nice to build every push to master to identify if anyone had broken the build. In theory waiting for a green build before merging should prevent this, but sometimes the merge goes badly or you are just trying to fix that one little thing that couldn't possibly break any tests...

martinjvickers commented 7 years ago

This really made me laugh this morning having just seen this feature notification when I got to work. Yesterday I did exactly what you described, three pushes to a repo within a minute or two due to spelling mistakes. This is a really great feature.

jamcat22 commented 7 years ago

I feel like we need some sort of status symbol within Travis' GitHub integration, especially with PRs. As of right now, GitHub shows the builds as failing, when it should show a "canceled" symbol or no symbol at all.

eliaskosunen commented 7 years ago

@jamcat22 I think GitHub only has three statuses for CI and such: success, pending and failure. It'd have to be an update on GH's end to provide support for this.

joshk commented 7 years ago

@martinjvickers haha, glad to hear the feature helped :)

@jamcat22 sadly @eliaskosunen is right, GitHub doesn't offer a means for us to mark something as canceled. I can raise this with them, but I'm not sure when/if we will hear back.

bsipocz commented 7 years ago

@jamcat22 @joshk - Correct me if I see it wrong, but this auto cancellation only has any effect when there is another commit in the given branch of the PR. So then how could its (the PR’s) status be failed rather than pending? Or you mean the individual status of the commit?

jamcat22 commented 7 years ago

@bsipocz I'm talking about the individual status of the commit.

ewang commented 7 years ago

Will this also work for force pushes?

IgorMinar commented 7 years ago

Great work, except I'd prefer if there was an option to enable this feature only for force pushes.

I often do work by pushing small commits to my PR branch frequently without even waiting for the results of the current run. This way I get to move quickly and can see build history to find out where I broke the build.

On the other hand, if I ever force-push to my PR branch, that is a clear signal that I don't care about the old commits and therefore any in progress builds for these commits should be canceled.

I believe this is exactly how CircleCI implements this feature a while back: https://discuss.circleci.com/t/auto-cancel-on-next-build/832/12

joshk commented 7 years ago

Hey @IgorMinar

Hmmmm, that is an interesting idea. I will think this over more as I think it should be the default behaviour (as it is likely the other commits won't be accessible)

Thanks a bundle for the feedback!

akaralar commented 7 years ago

I would love to see an option to cancel currently running builds as well. It makes sense if you:

Have 2 or 1 concurrent builds (and don't have the cash for a bigger plan)
Have 4-5 developers working on the same repo,
Work in WIP PR's (PR's are opened when the developer starts implementing a feature so that peers can peek and review continuously)
Builds take a long time
Can't merge before travis gives a checkmark.

Working this way, there might be a lot of queued builds, but we only care about the latest commit on a PR

We are using a deduplicator for this now and it cancels currently running builds too. I hope this feedback finds you well :)

peternewman commented 7 years ago

This didn't seem to work for https://travis-ci.org/OpenLightingProject/ola/builds/217252701 and https://travis-ci.org/OpenLightingProject/ola/builds/217252811 which were just an open PR and update branch away, despite having already turned on the skip on PRs option.

lneuhaus commented 7 years ago

This didnt really seem to work for me (repository lneuhaus/pyrpl), or its not obvious what this thing does.

My builds take a very long time (>1h), and have to run sequentially. E.g., while build 1.1 is running, with 1.2-4 in the waiting list, I keep on submitting more commits to the pullrequest. After that, the waiting list reads 1.2-4, 2.1-4, 3.1-4 for example.

When I cancel build 1.1, build 1.2 is executed (I would prefer 3.1, but I can live with that behavior). But even worse, if I cancel 1.4, 1.3, and the running 1.2, the next build is 2.1 instead of 3.1. I don't see how this makes sense.

joshk commented 7 years ago

@akaralar Thanks for the feedback. We plan to support the ability to cancel currently running Jobs in the near future.

@peternewman Sorry to hear that, could you email support@travis-ci.com and we can look into it.

@lneuhaus Sorry to hear that you are having issues with this feature. Is the description at the top, or where you enable this feature, not clear? Could you email support@travis-ci.com explaining the problem further and we can look into it. Thanks a bundle!

LefterisJP commented 7 years ago

@joshk Quick question about this feature. What is the expected behavious when auto cancelation is ON, there is no currently running build in a branch and you force push into it by amending the last commit but not adding any new commits?

Will a build properly run?

joshk commented 7 years ago

Hi @LefterisJP, sorry, I'm not sure I understand. You mention force pushing by amending the last commit, was this last commit also pushed to GitHub? Travis isn't running any builds, is that expected?

LefterisJP commented 7 years ago

Yeah so the last commit was pushed and built succesfully. Then when you amend it locally and force push to github again you would expect Travis to generate a build right even if autocancelation is on?

We had an issue where Travis was not running a build, but I think we found the problem and it's not related to autocancelation.

dwsteele commented 7 years ago

I would like to be able to disable this for certain branches, like integration and master. For all other branches it is certainly the way to go. Any way to do that?

joshk commented 7 years ago

@dwsteele Thanks for the feedback. There currently isn't a way to do that, but I do like that idea. I will note that for the second round of development.

hughperkins commented 7 years ago

Question: what will happen to the place of jobs in the queue?

if queued jobs will get pulled, and the new jobs get added to the tail of the queue, I'd rather just leave my jobs in the queue, at least get feedback earlier, even if for older jobs
on the other hand, if you replace whichever job is nearest the head of the queue with the latest commit, that seems open to abuse

For now, I'm just posing the question :-)

joshk commented 7 years ago

@hughperkins the feature is fairly simple in that it will cancel jobs for the entire Build, and add the new Build to the end (tail) of the queue.

travis-ci / beta-features

Auto Cancellation : Get faster results by only testing your most recent build #8