Job stuck in "Pending Completion"

[davidbyoung]

I haven't made any changes to the way my repository reports coverage info. I've noticed that my jobs are all stuck with "Pending completion". I am not using parallel builds, and am unsure how to fix this. For reference, here is my project: https://coveralls.io/github/aphiria/aphiria.

[davidbyoung]

hmm, on the latest build, this appears to have disappeared. It was stuck like this for quite some time on my PR builds, but when I merged the PR, it resolved. Please disregard.

[afinetooth]

Hi @davidbyoung.

Thanks for reporting your issue, and thanks for the update. I'm glad it cleared itself.

In case it's helpful, I looked at stats for your builds and, at least for the past 7 days, the avg build time is ~1min, with the longest build time being 2min. So what that means, practically, is that you may see "pending completion" at your job/build page at Coveralls.io from the point you receive it in your API response until the job/build is complete (so far 1-2 min), but should shouldn't be seeing it for any longer than that. If you did, let me know.

Additional info on long-running build(s) for your repo:

• I looked at your stats for the last 30-days and they are the same: avg build time = 1min; longest build time = 2min.

• That is the same for the last 90-days with one exception:

  • On 9/17, you had three (3) sequential builds between 10:05-10:35a PDT that were slower than normal, with one build we consider an anomaly, with a 45min build time.

    • That build had a normal processing time (1:26), but, what we call its "external wait time", was 44:02, representing 96.8% of the total build time.
    • One-off performance issues like this happen from time-to-time, typically cluster around a particular day/time, and the root cause is usually some external network factor, commonly rate limiting (for your account/token) from the GItHub API. That would be my guess on that one.
    • Update: While our production bug tracking system only goes back 30-days, I see a repeating pattern for builds from that repo for PR 313 (which your offending build is as well): 406 (Octokit::NotAcceptable), re: "Diff to large." It means GitHub can't fulfill the underlying GET request because your PR's diff exceed the limit for your account (which may be a universal limit).

    Here's the full error message from GitHub:

    Octokit::NotAcceptable · GET https://api.github.com/repos/aphiria/aphiria/pulls/313: 406 - Sorry, the diff exceeded the maximum number of files (300). Consider using 'List pull requests files' API or locally cloning the repository instead. Error summary: resource: PullRequest field: diff code: too_large // See: https://docs.github.com/rest/pulls/pulls#list-pull-requests-files

    Files changed in that PR: 390:

    

Re: that 406 issue: It's something we started seeing in Apr of this year and are trying to address, but it requires significant code changes on our end, has been rated indeterminate complexity, and has not been added to a sprint yet. The only current workaround is to either forego coverage stats on your "big" PR, or break it into smaller pieces to get stats on each piece.

[davidbyoung]

Thank you so much for the detailed response! I did notice that my PR was stuck in this pending state for most of the time it was open, and not just for a short period after the build process. However, I do not see it anymore now that the PR was merged into the 1.x branch.

[afinetooth]

You're welcome. And now that I think about it, that makes complete sense, since, we would not have been able to fully process and perform an aggregate coverage calculation on your build(s) for that PR as long as we could not obtain the full diff (to perform a coverage diff) from the GitHub API.

So, for the benefit of anyone else experiencing a longstanding "Pending Completion" on builds for a certain PR, this is most likely your issue. We have a dev ticket in queue to pass through end user account-related GitHub API errors/messages, around such things as exceeding rate limit or this diff size limit, so they appear in PR Comments, or status checks, or at least Coveralls Build Pages, to help users understand what's going on in these fairly rare circumstances.