No comments on pull requests from coveralls in emqx/emqx

[id]

We've stopped receiving comments from coveralls bot for quite some time in emqx/emqx repo. The last build that I could find with a comment to the corresponding PR is this one.

• I have invited the @coveralls user to emqx/emqx with Role: Write
• I have enabled the "LEAVE PR COMMENTS?" setting in the "Pull Request Alerts" area in my Repo Settings page inside the Coveralls app.

The only difference from coveralls action requirement is that workflow in question is not triggered by pull_request event, but rather is called by parent workflow. Which, in turn, is triggered by the pull_request event.

Here is my PR where I tried to troubleshoot this: https://github.com/emqx/emqx/pull/12987. I've switched from using curl to coverallsapp/github-action, and seems like it worked, but still no comment in the PR: https://github.com/emqx/emqx/actions/runs/9004245864/job/24739939195?pr=12987#step:2:56

 API Response: {"done":true,"url":"https://coveralls.io/builds/67376894","jobs":0}

At least, build number in the done response matches number in responses for parallel builds like this one.

I also noticed that coveralls does not refresh branch list, it shows some weird branches which are not even there, and may be even tags.

Could you help me to understand what is the problem?

[afinetooth]

Hi @id.

The explanation of this issue is complicated, and somewhat confusing due to a lot of seemingly related but ultimately irrelevant datapoints.

But the fix is simple, so let's start there and, if you want any further background, we can get into that.

Missing base build -> No PR Comment

Basically, in order to send a PR Comment, Coveralls needs to be able to find a PR build's base build---which is to say a coverage report for its base build---to which it compares the current build to derive a report showing the change in coverage between the two relevant commits.

That change in coverage is the key content of the PR Comment and when Coveralls can't find the base build, it can't derive the contents of the PR Comment and, as a result, it won't send one.

We are working on improvements to the UX of this scenario, which include a PR Comment with an exception message and a link to troubleshooting steps, but for the time being, your clue that this is your issue is that your PR build will read " FIRST BUILD" in the header of its build info table.

For example: https://coveralls.io/builds/67917577

Problem

While there appear to be a few issues with your builds, leading to misdirection, like the fact that the branch name is missing from your push builds:

Which points to a "git hash" missing from your coverage reports that Coveralls integrations expect to pick up from the CI environment, but since your coverage reports still contain the minimum required git data---commit_sha:

This isn't your issue because the presence of commit_sha allows Coveralls to find your PR builds' base builds.

Your problem is that your base builds were never completed.

In other words, your PR builds' base builds are in a state known as "pending completion" and, as a result, they do not appear in your official Build History and cannot be used for coverage calculations or comparisons.

Here is an example using the same PR build from above: https://coveralls.io/builds/67917577

Here is the PR info we have from GitHub for that build:

PR Info:

{
  "title": "ci: OTP 26 (26.2.5-1) for docker images",
  "head": {
    "sha": "c41be7ee22b310e46786116e20088b84bb528a3e",
    "label": "zmstone:0605-otp26-for-docker-images",
    "ref": "0605-otp26-for-docker-images"
  },
  "base": {
    "sha": "ae0379f974a25f46ee7ba81d44c3dde68f18956e",
    "label": "emqx:release-57",
    "ref": "release-57"
  }
}

Where "base"["sha"] is the commit associated with your base build, and the one Coveralls is looking for, for comparison.

And that build exists:

Build.find(67917577).repo
=> #<GithubRepo id: 229018, name: "emqx/emqx" [...]

repo.builds.where(commit_sha: "ae0379f974a25f46ee7ba81d44c3dde68f18956e").first
=> #<Build id: 67904199, repo_id: 229018, [...] service_name: "github", [...] service_number: "9383783553", service_event_type: "push", [...] state: "pending" [...]

And here's the Coveralls build page for that build: https://coveralls.io/builds/67904199

The problem is that it's pending completion---it's not finished / not completed, and its coverage was never calculated:

I'm not sure why this is, but the evidence points to these builds never fully uploading all of their expected parallel uploads (in your case 110 uploads), and, more to the point, never sending Coveralls the "close parallel build" signal, which means no steps in your Ci config ever called the (close) parallel build webhook.

Using this example, here's the PR build with 109 uploads (per my Admin info), which was completed: https://coveralls.io/builds/67917577

And here's the base build, with only 105 uploads, and never completed: https://coveralls.io/builds/67904199

To back that up, here's another example:

PR build with 109 uploads, completed: https://coveralls.io/builds/67881752

PR Info:

{
  "title": "fix(resource manager): force kill process if stuck when stopping/removing",
  "head": {
    "sha": "f17aefe3d7746da91c917907536a7f6ae2b52317",
    "label": "thalesmg:force-stop-connector-r57-20240604",
    "ref": "force-stop-connector-r57-20240604"
  },
  "base": {
    "sha": "cda6d5f63636199932d9c1015d690b011c46e243",
    "label": "emqx:release-57",
    "ref": "release-57"
  }
}

Base build with 105 uploads, never completed: https://coveralls.io/builds/67881435

Fix

I assume that if you can figure out what's happening with those CI builds (from their build logs) in terms of why the uploads are stopping at 105, and why the (close) parallel build signal is never being sent, and fix that, you'll fix this problem.

If you have trouble figuring that out, please share the relevant portions of your CI build log(s), and, even better, if you can invoke debug or verbose mode that will help us both diagnose what's happening more easily. If you're using the Coveralls GitHub Action, just use the debug input option:

    - name: Coveralls Parallel Upload Step
      uses: coverallsapp/github-action@v2
      with:
        flag-name: whatever
        parallel: true
        debug: true

And:

    - name: Coveralls Close Parallel Build Step
      uses: coverallsapp/github-action@v2
      with:
        parallel-finished: true
        debug: true

[id]

Hi @afinetooth.

Thanks a lot for the detailed explanation, very helpful! I'll troubleshoot the problem on our end, now that I know where to look.