API returning Same `Commit Hash` YET Different Coverage Percentage and Date

[ErnestFistozz]

Hi,

I am trying to collect coverage from one of the repository in the Apache Foundation. The coverage data is specifically for the master branch. The issue Branch Coverage Through API describes how to retrieve the desired response. My endpoint is:

https://coveralls.io/github/apache/rocketmq.json?page=1&branch=master

Below is the response i receive. However, the response is not correct because the created_at AND covered_percent properties changes YET the commit_sha property remains the same. That is definitely a BUG because a git commit hash cannot be the same for different commits. Each hash is unique.

I am particularly referring to first element to the fourth element in the builds array i.e builds[0], builds[1], builds[2] and builds[3].

{
"page": 1,
"pages": 45,
"total": 445,
"builds": [
{
"created_at": "2022-06-28T06:09:43Z",
"url": null,
"commit_message": "Merge branch 'develop'",
"branch": "master",
"committer_name": "XXXX",
"committer_email": "XXXX",
"commit_sha": "2622ba3a7f77476303c21405cb7da46f73907599",
"repo_name": "apache/rocketmq",
"badge_url": "https://s3.amazonaws.com/assets.coveralls.io/badges/coveralls_51.svg",
"coverage_change": 0.009,
"covered_percent": 51.37303809680684
},
{
"created_at": "2022-05-28T03:34:26Z",
"url": null,
"commit_message": "Merge branch 'develop'",
"branch": "master",
"committer_name": "XXXX",
"committer_email": "XXXX",
"commit_sha": "2622ba3a7f77476303c21405cb7da46f73907599",
"repo_name": "apache/rocketmq",
"badge_url": "https://s3.amazonaws.com/assets.coveralls.io/badges/coveralls_51.svg",
"coverage_change": -0.2,
"covered_percent": 51.3636256676942
},
{
"created_at": "2022-04-28T05:05:49Z",
"url": null,
"commit_message": "Merge branch 'develop'",
"branch": "master",
"committer_name": "XXXX",
"committer_email": "XXXX",
"commit_sha": "2622ba3a7f77476303c21405cb7da46f73907599",
"repo_name": "apache/rocketmq",
"badge_url": "https://s3.amazonaws.com/assets.coveralls.io/badges/coveralls_52.svg",
"coverage_change": -0.03,
"covered_percent": 51.58246464456316
},
{
"created_at": "2022-03-28T03:34:27Z",
"url": null,
"commit_message": "Merge branch 'develop'",
"branch": "master",
"committer_name": "XXXX",
"committer_email": "XXXX",
"commit_sha": "2622ba3a7f77476303c21405cb7da46f73907599",
"repo_name": "apache/rocketmq",
"badge_url": "https://s3.amazonaws.com/assets.coveralls.io/badges/coveralls_52.svg",
"coverage_change": 0.03,
"covered_percent": 51.6154081464574
},
{
"created_at": "2022-03-23T15:20:52Z",
"url": null,
"commit_message": "Merge pull request #4 from apache/master\n\nmerge from Apache Master",
"branch": "master",
"committer_name": "GitHub",
"committer_email": "noreply@github.com",
"commit_sha": "8f5c2610d6005328358d2a9cd2bdffa2f1b91a9c",
"repo_name": "apache/rocketmq",
"badge_url": "https://s3.amazonaws.com/assets.coveralls.io/badges/coveralls_52.svg",
"coverage_change": 0.03,
"covered_percent": 51.9111424541608
}
]
}

NB : I have redacted the author's email and name

[afinetooth]

Hi @ErnestFistozz.

I think that, in this case, this is correct.

If you notice the project's recent builds:

The last four (4) appear to have been triggered by a once-monthly cron job. I would venture to say that, while this triggered a new build in CI, there were no changes to the repo (on master) and therefore, each new build was created using the same commit sha.

This tracks with the fifth (5th) most recent build (Build: 6338), the fifth (5th) in your results, having a different commit sha and different information from Github (commit, committer, type, via, etc).

As for the created_at field, that pertains to the build, which is a different build in CI, so no surprise there.

But you have a good point about the covered_percent being different, which doesn't make a lot of sense for what should be identical builds.

That said, there are a number of reasons why test coverage results could be different for the identical codebase with no changes to code or tests. In this case, one might be time-sensitive tests.

Another thing I noticed that could account for unexpected coverage changes is that, while Builds 6338-6972 each have two (parallel) jobs:

Example: https://coveralls.io/builds/47730656

Build 7257 only has one: https://coveralls.io/builds/50412252

Which could indicate a failed build, or something else, like a change to CI config.

[ErnestFistozz]

Hi @afinetooth

Thanks, I don't know why I didn't think of cron jobs. but you are absolutely correct it is a cron job.

Quick question though, when it comes to the number of reasons link, this points me to a tool called CodeCov. (which is another tool i am using). Does this mean CodeCov and Coveralls are the same from a design perspective.

[afinetooth]

Hi @ErnestFistozz. Yes, Codecov is a competitor of Coveralls in the test-coverage-as-a-service space. We've been around a bit longer and focus on open-source as part of our original mission. They have great documentation and cover that topic well.

[ErnestFistozz]

Completed!