How do I configure a push build in a Pull Request context?

[snackattas]

How do I configure a push build in a Pull Request context?

https://docs.coveralls.io/build-types#recommended-ci-configurations

how would I implement this strategy with this tool? It seems there's no options to configure whether you want your upload to count as a pull request build or push build

[afinetooth]

Hi @snackattas. I see your emails to support@coveralls.io as a paying subscriber so I will answer those in email, but I wanted to respond here as well:

How do I configure a push build in a Pull Request context? https://docs.coveralls.io/build-types#recommended-ci-configurations

All you need to do is configure your CI service to send Coveralls coverage reports for push commits (in generals, or just on your PR branches).

So using GitHub Actions as the example, that would look like this:

on:
  push:

Which just means "build all push builds."

You can further specify branch patterns if you have a known pattern for how you name PR branches.

Alternatively, you can build all push commits, and all pull_request commits targeting your default branch (ex. main) like this:

on:
  push:
  pull_request:
    branches:
      - main

More info on how Coveralls handles push builds on your PR branches per our CI config recommendations:

Since we started making changes to the way we handle "drifted" pull request builds in Coveralls (pull request builds whose base commit is no longer the head of the PR's target branch) in early Nov, we've changed the way Coveralls handles push builds that belong to PRs:

• We now treat any new push build that belongs to a PR as though it were the pull request build for the current head of your PR branch.
• We do this by checking each push build to see if it belongs to an open PR, and, if it does, we render it as if it were for the latest pull_request commit.
• But, we also send accompanying status updates to the relevant push commit at GitHub, and we include two different status updates to your PR (if status updates are enabled):
  • coverage/coveralls (push)
  • coverage/coveralls (pull_request)
• We do that so you can distinguish the coverage details for the push commit vs the coverage details for the PR at that point.
• If you are also sending us coverage reports for pull_request builds on your PR branches, those will essentially be a duplicate of the build above, but those pull_request builds remain susceptible to becoming "drifted", whereas the pull request builds we derive from comparing the latest push commit on your PR branch with the PR's base commit (on your target branch) should be an accurate three-dot-diff comparison, resulting in an accurate coverage report for your PR (at that point in time).

Sorry if I lost the thread in all those details. I though you might want to know the rest, once you learned that we are actually treating some of your push builds as though they were pull_request builds.

As I say, I will follow up in email, but let me know if, for yourself, or for others who may be having the same issue, I can answer some more questions about this use case.

[snackattas]

Alternatively, you can build all push commits, and all pull_request commits targeting your default branch (ex. main) like this:

Hey @afinetooth , one thing is, I don't use github actions, I use Codefresh.

So how would I specify these configurations with just the coverage-reporter tool? That was one of my main points of confusion.

But, we also send accompanying status updates to the relevant push commit at GitHub, and we include two different status updates to your PR (if status updates are enabled): coverage/coveralls (push) coverage/coveralls (pull_request)

On this point, I have been using the coverage-reporter, but only ever see pull_request builds posted back to the PR.

When my PR is merged into master, I have a job that runs the same CI unit tests/coverage reporter on master, and that posts back as a push build.

But I have not seen the push build posted back to a PR. How do I get this to happen? I want that elusive 3 dot coverage

I though you might want to know the rest, once you learned that we are actually treating some of your push builds as though they were pull_request builds.

How can I tell if this is the case?

[afinetooth]

I don't use github actions, I use Codefresh.

So how would I specify these configurations with just the coverage-reporter tool? That was one of my main points of confusion.

If you don't use Github Actions or CircleCI, then you'll want to bypass both of those official extensions for those services and use the Universal Coverage Reporter directly.

These usage examples, plus the section below it, with the output you'll see from invoking coveralls --help will show you the flag to use that are equivalent to the input options I've mentioned for the GitHub Action.

For general CI configuration, like telling your CI that you want to build push builds on your PR branches, that's something I don't have the expertise to give you, since it's based on Codefresh syntax. However, this Codefresh doc seesm to suggest that that use case might be the default behavior of Codefresh:

Building branches automatically By default, Codefresh connects to your Git provider and does the following:

1. Auto-builds every new commit that happens in master or any other branch
2. Auto-builds every new branch when it is created

You can change the default behavior so that it matches your own workflow using extra Git triggers.

You don’t have to do anything special to set up this communication between Codefresh and your Git provider. It was set up automatically when you connected your Codefresh account to your Git provider.

And this section on Git Triggers seems to document how to trigger Codefresh pipeline on push or pull_request events, which they list among their git trigger events.

To conform to our Recommended CI Config, "just build push commits on all branches," it looks like the relevant event is Push commits (which also seems to be the default setup).

To also build on pull_request events, you probably, minimally, want to use the Pull Request synchronized trigger.

But again, you probably don't need to do that, and can probably see how it goes with just building all your push commits before adding more trigger events.

[afinetooth]

But, we also send accompanying status updates to the relevant push commit at GitHub, and we include two different status updates to your PR (if status updates are enabled): coverage/coveralls (push) coverage/coveralls (pull_request)

On this point, I have been using the coverage-reporter, but only ever see pull_request builds posted back to the PR.

Can you please share the Coveralls Build URL for such as PR build? I can see what status updates we sent and how that aligns with your repos settings for status updates. Then we can determine if it's a config thing or a bug.

When my PR is merged into master, I have a job that runs the same CI unit tests/coverage reporter on master, and that posts back as a push build.

That behavior, as described, is expected. After you merge a PR into master, it results in a new push commit on master, which, if you're building all push commits (Codefresh default behavior), and sending coverage reports for them to Coveralls, we'll have a new push build for you on master in Coveralls.

That's what you want, because if that commit becomes the merge base for a new PR, that coverage report (ie. Coveralls "base build") will be needed to calculate three-dot diffs with the new PR's HEAD commit(s).

But I have not seen the push build posted back to a PR. How do I get this to happen? I want that elusive 3 dot coverage

That push build ( on master) does not get posted back to a PR. Because it's on master it's no longer associated with a pull request build, until such time as it may become the base build of a new PR.

Rest assured that if you create a new PR from any push commit on master, Coveralls will look for that push commit's coverage report to compare it with the commits in your new PR.

So it is good, and correct, that you are building push commits on your master branch because any of those could become the "base build" for a new PR (Where "base build" is Coveralls parlance for the coverage report for those push commits).

[afinetooth]

I though you might want to know the rest, once you learned that we are actually treating some of your push builds as though they were pull_request builds.

How can I tell if this is the case?

You can trust that we will generate a (Coveralls) pull request build for any coverage reports you send us for push commits that belong to any of your open PRs.

But you can verify that by following the link in the API response from Coveralls in your CI build log to the new job or build at Coveralls and reviewing the associated build.

Example:

 

I don't want to use one of your builds as an example since your repos are private, but using one of our PRs for Coveralls:

These are our Repo Settings:

We are configured to receive:

1. A status update for each build
2. Status updates for each job (or subproject) in the build (the children of a parallel build)
3. A status update for any PR patch
4. Status updates for pull request builds showing the "(pull_request)" context

Note: _(4) is a legacy thing. When we started generating pull request builds for push builds belonging to PRs, we changed our legacy context value (coverage/coveralls), which previous applied for either push or pull_request builds, and allowed for an extra modifier to further distinguish the context: coverage/coveralls (push) and coverage/coveralls (pull_request). The problem is that a lot of legacy projects had branch protection rules based on our old context (coverage/coveralls) so, in order not to break those, we made the ability to get that extra context differentiation into a setting. We recommend enabling it if you don't have legacy branch protection rules that rely on the coverage/coveralls context._

Another note: Because we are rendering your push build as though it was a pull request build, we send two different associated status updates:

1. One for the original push commit belonging to the PR (coverage/coveralls (push)); and
2. One for the Coveralls pull request build that we generated for it, which would have otherwise been synchronized as a new pull_request build in CI. We send that status update to the push commit and to the associated PR (coverage/coveralls (pull_request))

Based on our status update settings this PR shows the status updates we receive as a result:

I am primarily highlighting the difference between the two relevant status updates:

• coverage/coveralls (push) - Which we send for the last push commit of the PR
• coverage/coveralls (pull_request) - Which we send for the pull request coverage report we generate from that last push commit.

One more note on coverage/coveralls (pull_request): _When we generate a Coveralls pull request build for a given push commit to an open PR, we generate a coverage report, and send a status update, for what would be an ideal pull_request build with a three-dot-diff comparison. Because we do this, we don't actually need you to send us a coverage report for the actual pull_request commit, which could possibly be faulty/drifted anyway). Our pull request build (as a three-dot diff comparison) is derived from doing what we call a coverage diff between the coverage report for the PR HEAD (the latest push commit to the PR) and the coverage report for the PR base (the push commit on master that is the PR's base commit)._

Hope that makes sense.

If you configure your status updates something like ours, you should get similar results.