coveralls frequently gets "stuck"

[jodh-intel]

coveralls is being run from the 16.04 Jenkins node. Even though the builds pass under Jenkins and Jenkins triggers the coveralls run (which also passes), the github PR page never updates for the coveralls test.

Examples:

• https://github.com/clearcontainers/runtime/pull/604
• https://github.com/clearcontainers/runtime/pull/619
• https://github.com/clearcontainers/runtime/pull/625
• https://github.com/clearcontainers/runtime/pull/629

[jodh-intel]

This is still happening. See for example https://github.com/clearcontainers/runtime/pull/729.

[jodh-intel]

Another stuck PR: https://github.com/clearcontainers/runtime/pull/760

The coveralls report is: https://coveralls.io/jobs/30598315

Looking back to before we switched to Jenkins (when we triggered coveralls from travis-ci) and comparing the above report with an old one (https://coveralls.io/builds/11704922), there are a few differences:

• Previously, the "committed by" user was always web flow.
• Previously, the "build type" was "push travis-ci" whereas now it is just "push".

Here's are some examples of successful (non-"stuck") coveralls build titles before we switched to Jenkins:

• COVERAGE INCREASED (+7.5%) TO 63.889% (committed by web-flow)
• COVERAGE REMAINED THE SAME AT 53.451% (committed by web-flow)
• COVERAGE DECREASED (-7.8%) TO 33.744% (committed by web-flow)

I can't find any coveralls reports in the pre-Jenkins era that don't conform to the above pattern. But if we look at the titles of "stuck" Jenkins+coveralls reports:

• COVERAGE DECREASED (-0.6%) TO 96.56% (committed by sboeuf)
• COVERAGE REMAINED THE SAME AT 97.196% (committed by sboeuf)
• COVERAGE REMAINED THE SAME AT 97.208% (committed by jodh-intel)
• FIRST BUILD ON FIX-LIST--CC-ALL AT 96.496% (committed by jodh-intel)
• FIRST BUILD ON UPDATE-ARTEFACTS AT 95.448% (committed by sboeuf)
• FIRST BUILD ON UNIT-TEST-COVERAGE-FIXES AT 96.799% (committed by sboeuf)

Prior to switching to Jenkins, the title never mentioned "FIRST BUILD" fwics. It appears that coveralls is considering the committer + branch name and if it hasn't seen either the committer or the branch before (or possibly that it hasn't seen the committers repo before), it will create a new baseline unit-test coverage figure for that combination. If so, that really isn't what we want.

But still, why no github updates? Could it be some sort of permissions issue where coveralls is attempting to update the PR status as the committer user?

[jodh-intel]

Any thoughts on this @sboeuf?

[sboeuf]

@jodh-intel nope, this needs to be further investigated.