⚠️ Internal server error. Please contact Coveralls team. / report submission fails for external contributors

niccokunzmann commented 2 weeks ago

Dear Coveralls team,

I got this message from the coverallsapp/github-action:

⚠️ Internal server error. Please contact Coveralls team.

Since we are facing this problem for a while now and thought we have found a fix (which now is not a fix), I would like to contact you about this. I wonder: How can this be approached?

Run coverallsapp/github-action@v2
  with:
    flag-name: run-py38
    parallel: true
    file: coverage.xml
    allow-empty: true
    github-token: ***
    coveralls-endpoint: https://coveralls.io/
    debug: false
    measure: false
    fail-on-error: true
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.18/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib
Run mkdir -p ~/bin/
  mkdir -p ~/bin/
  cd ~/bin/
  curl -sLO https://github.com/coverallsapp/coverage-reporter/releases/latest/download/coveralls-linux.tar.gz
  curl -sLO https://github.com/coverallsapp/coverage-reporter/releases/latest/download/coveralls-checksums.txt
  cat coveralls-checksums.txt | grep coveralls-linux.tar.gz | sha256sum --check
  tar -xzf coveralls-linux.tar.gz
  rm coveralls-checksums.txt
  echo ~/bin >> $GITHUB_PATH
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.18/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib
coveralls-linux.tar.gz: OK
Run coveralls report    --allow-empty   coverage.xml 
  coveralls report    --allow-empty   coverage.xml 
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.18/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib
    COVERALLS_DEBUG: false
    COVERALLS_CARRYFORWARD_FLAGS: 
    COVERALLS_FLAG_NAME: run-py38
    COVERALLS_PARALLEL: true
    COVERALLS_ENDPOINT: https://coveralls.io/
    COVERALLS_GIT_BRANCH: 
    COVERALLS_GIT_COMMIT: 
    COVERALLS_REPO_TOKEN: ***
    COVERALLS_COMPARE_REF: 
    COVERALLS_COMPARE_SHA: 
    COVERALLS_SOURCE_HEADER: github-action

⠀⠀⠀⠀⠀⠀⣿
⠀⠀⠀⠀⠀⣼⣿⣧⠀⠀⠀⠀⠀⠀⠀ ⣠⣶⣾⣿⡇⢀⣴⣾⣿⣷⣆ ⣿⣿⠀⣰⣿⡟⢸⣿⣿⣿⡇ ⣿⣿⣿⣷⣦⠀⠀⢠⣿⣿⣿⠀⠀⣿⣿⠁⠀⣼⣿⡇⠀⢀⣴⣾⣿⡷
⠶⣶⣶⣶⣾⣿⣿⣿⣷⣶⣶⣶⠶  ⣸⣿⡟ ⠀⢠⣿⣿⠃⠈⣿⣿⠀⣿⣿⢠⣿⡿⠀⣿⣿⣧⣤⠀⢸⣿⡇⣠⣿⡿⠀⢠⣿⡟⣿⣿⠀⢸⣿⡿⠀⠀⣿⣿⠃⠀⢸⣿⣧⣄
⠀⠀⠙⢻⣿⣿⣿⣿⣿⡟⠋⠁⠀⠀ ⣿⣿⡇⠀ ⢸⣿⣿⠀⣸⣿⡟⠀⣿⣿⣾⡿⠁ ⣿⣿⠛⠛⠀⣿⣿⢿⣿⣏⠀⢀⣿⣿⣁⣿⣿⠀⣾⣿⡇⠀⢸⣿⡿⠀⠀⡀⠙⣿⣿⡆
⠀⠀⢠⣿⣿⣿⠿⣿⣿⣿⡄⠀⠀⠀ ⠙⢿⣿⣿⠇⠈⠿⣿⣿⡿⠋⠀⠀⢿⣿⡿⠁⠀⢸⣿⣿⣿⡇⢸⣿⣿⠀⣿⣿⣄⣾⣿⠛⠛⣿⣿⢠⣿⣿⣿ ⣼⣿⣿⣿ ⣿⣿⡿⠋⠀
⠀⢀⣾⠟⠋⠀⠀⠀⠙⠻⣷⡀⠀⠀

  v0.6.12

📄 Using coverage file: coverage.xml
⭐️ Running in parallel mode. You must call the webhook after all jobs finish: `coveralls done --build-number 9829914780`
  ·job_flag: run-py38
🚀 Posting coverage data to https://coveralls.io/api/v1/jobs
⚠️ Internal server error. Please contact Coveralls team.
Error: Process completed with exit code 1.

See also:

niccokunzmann commented 2 weeks ago

Update: Python 3.10 worked on one build and Python 3.9 did not....

Hm. Python 3.10 works:

Run coveralls report    --allow-empty   coverage.xml 

⠀⠀⠀⠀⠀⠀⣿
⠀⠀⠀⠀⠀⣼⣿⣧⠀⠀⠀⠀⠀⠀⠀ ⣠⣶⣾⣿⡇⢀⣴⣾⣿⣷⣆ ⣿⣿⠀⣰⣿⡟⢸⣿⣿⣿⡇ ⣿⣿⣿⣷⣦⠀⠀⢠⣿⣿⣿⠀⠀⣿⣿⠁⠀⣼⣿⡇⠀⢀⣴⣾⣿⡷
⠶⣶⣶⣶⣾⣿⣿⣿⣷⣶⣶⣶⠶  ⣸⣿⡟ ⠀⢠⣿⣿⠃⠈⣿⣿⠀⣿⣿⢠⣿⡿⠀⣿⣿⣧⣤⠀⢸⣿⡇⣠⣿⡿⠀⢠⣿⡟⣿⣿⠀⢸⣿⡿⠀⠀⣿⣿⠃⠀⢸⣿⣧⣄
⠀⠀⠙⢻⣿⣿⣿⣿⣿⡟⠋⠁⠀⠀ ⣿⣿⡇⠀ ⢸⣿⣿⠀⣸⣿⡟⠀⣿⣿⣾⡿⠁ ⣿⣿⠛⠛⠀⣿⣿⢿⣿⣏⠀⢀⣿⣿⣁⣿⣿⠀⣾⣿⡇⠀⢸⣿⡿⠀⠀⡀⠙⣿⣿⡆
⠀⠀⢠⣿⣿⣿⠿⣿⣿⣿⡄⠀⠀⠀ ⠙⢿⣿⣿⠇⠈⠿⣿⣿⡿⠋⠀⠀⢿⣿⡿⠁⠀⢸⣿⣿⣿⡇⢸⣿⣿⠀⣿⣿⣄⣾⣿⠛⠛⣿⣿⢠⣿⣿⣿ ⣼⣿⣿⣿ ⣿⣿⡿⠋⠀
⠀⢀⣾⠟⠋⠀⠀⠀⠙⠻⣷⡀⠀⠀

  v0.6.12

📄 Using coverage file: coverage.xml
⭐️ Running in parallel mode. You must call the webhook after all jobs finish: `coveralls done --build-number 9829914780`
  ·job_flag: run-py310
🚀 Posting coverage data to https://coveralls.io/api/v1/jobs
---
✅ API Response: {"message":"Coverage for parallel build uploaded","url":"https://coveralls.io/builds/68499778"}
- 💛, Coveralls

Python 3.9 fails:

Run coveralls report    --allow-empty   coverage.xml 
  coveralls report    --allow-empty   coverage.xml 
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.19/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.19/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.19/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.19/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.19/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.19/x64/lib
    COVERALLS_DEBUG: false
    COVERALLS_CARRYFORWARD_FLAGS: 
    COVERALLS_FLAG_NAME: run-py39
    COVERALLS_PARALLEL: true
    COVERALLS_ENDPOINT: https://coveralls.io/
    COVERALLS_GIT_BRANCH: 
    COVERALLS_GIT_COMMIT: 
    COVERALLS_REPO_TOKEN: ***
    COVERALLS_COMPARE_REF: 
    COVERALLS_COMPARE_SHA: 
    COVERALLS_SOURCE_HEADER: github-action

⠀⠀⠀⠀⠀⠀⣿
⠀⠀⠀⠀⠀⣼⣿⣧⠀⠀⠀⠀⠀⠀⠀ ⣠⣶⣾⣿⡇⢀⣴⣾⣿⣷⣆ ⣿⣿⠀⣰⣿⡟⢸⣿⣿⣿⡇ ⣿⣿⣿⣷⣦⠀⠀⢠⣿⣿⣿⠀⠀⣿⣿⠁⠀⣼⣿⡇⠀⢀⣴⣾⣿⡷
⠶⣶⣶⣶⣾⣿⣿⣿⣷⣶⣶⣶⠶  ⣸⣿⡟ ⠀⢠⣿⣿⠃⠈⣿⣿⠀⣿⣿⢠⣿⡿⠀⣿⣿⣧⣤⠀⢸⣿⡇⣠⣿⡿⠀⢠⣿⡟⣿⣿⠀⢸⣿⡿⠀⠀⣿⣿⠃⠀⢸⣿⣧⣄
⠀⠀⠙⢻⣿⣿⣿⣿⣿⡟⠋⠁⠀⠀ ⣿⣿⡇⠀ ⢸⣿⣿⠀⣸⣿⡟⠀⣿⣿⣾⡿⠁ ⣿⣿⠛⠛⠀⣿⣿⢿⣿⣏⠀⢀⣿⣿⣁⣿⣿⠀⣾⣿⡇⠀⢸⣿⡿⠀⠀⡀⠙⣿⣿⡆
⠀⠀⢠⣿⣿⣿⠿⣿⣿⣿⡄⠀⠀⠀ ⠙⢿⣿⣿⠇⠈⠿⣿⣿⡿⠋⠀⠀⢿⣿⡿⠁⠀⢸⣿⣿⣿⡇⢸⣿⣿⠀⣿⣿⣄⣾⣿⠛⠛⣿⣿⢠⣿⣿⣿ ⣼⣿⣿⣿ ⣿⣿⡿⠋⠀
⠀⢀⣾⠟⠋⠀⠀⠀⠙⠻⣷⡀⠀⠀

  v0.6.12

📄 Using coverage file: coverage.xml
⭐️ Running in parallel mode. You must call the webhook after all jobs finish: `coveralls done --build-number 9829914780`
  ·job_flag: run-py39
🚀 Posting coverage data to https://coveralls.io/api/v1/jobs
⚠️ Internal server error. Please contact Coveralls team.
Error: Process completed with exit code 1.

afinetooth commented 2 weeks ago

Hi @niccokunzmann. Thanks for reporting your issue.

I have submitted the behavior, but I have a few more questions:

Questions:

Are python 3.8 and 3.9 consistently not working, or do they sometimes work?
Flipside of (1): Do any jobs shown inconsistent behavior? Have they worked, then failed, or failed and then worked?
Is there any pattern, like, the first job to run always succeeds but the rest fail?
Can you please share the Coveralls URL for your project? (If you would rather keep your info private, please email to support@coveralls.io and reference this issue.)

niccokunzmann commented 2 weeks ago

Hi @afinetooth, thanks for taking this up!

Are python 3.8 and 3.9 consistently not working, or do they sometimes work?

I cannot say that, yet. When changes are made to https://github.com/collective/icalendar/pull/690, I might have more data. I cannot do it as I have write access. main is running now but the PR is from outside contributors.

Flipside of (1): Do any jobs shown inconsistent behavior? Have they worked, then failed, or failed and then worked?

They worked on main, they work when I created a PR. https://github.com/collective/icalendar/pull/690 is failing though.

Is there any pattern, like, the first job to run always succeeds but the rest fail?

I do not have a pattern yet, as there is only one failed run since we fixed it. There will be more, with patience. I restarted the job to see if my authorization might make a difference...

https://coveralls.io/builds/68499778

Indeed, this makes a difference. I am the maintainer and I am allowed to submit coverage. External contributors are not allowed.

Coveralls submits the report successfully to GitHub: https://github.com/collective/icalendar/pull/690#issuecomment-2217134821

I wonder why. I did not expect that. Is this expected behaviour?

Can you please share the Coveralls URL for your project? (If you would rather keep your info private, please email to support@coveralls.io and reference this issue.)

Is there some private URL/token for this or is it a human readable URL? Could you provide a template/example?

afinetooth commented 1 week ago

Hi @niccokunzmann. TBH I'm a little confused. Let me give some answers and then ask my questions:

Is there any pattern, like, the first job to run always succeeds but the rest fail?

I do not have a pattern yet, as there is only one failed run since we fixed it. There will be more, with patience. I restarted the job to see if my authorization might make a difference...

https://coveralls.io/builds/68499778

I noticed that this build succeeded, but that there was a prior CI run involved, from Jul 07, that left behind one, early submission of the coverage report with flag_name: PY310. Per my admin info about your uploads:

So, it looks like coverage reports were sent once for the build, originally, on Jul 07, then sent again, on Jul 09.

I'm not sure if both CI builds were successful, or if the first CI build failed and the second one succeeded, and I'm not sure if you're aware of that, or if that triggers any insights. I just wanted to mention it in case it does.

An open question for me here though is why the first coverage report for PY310 wasn't replaced by the second one, as seems to be the case for the rest of them. (Unless you know that only the PY310 report was sent during the first CI build?)

Indeed, this makes a difference. I am the maintainer and I am allowed to submit coverage. External contributors are not allowed.

I'm glad that worked for you, but just to clarify things from the Coveralls side, there is not necessarily anything that would prevent an external contributor from submitting coverage to Coveralls.io.

An external collaborator submitting a PR from a fork can expect to at least see a coverage report in the form of a PR Comment coming from Coveralls, as long as the PR they submitted was built in, and sent to Coveralls from, your CI service, with your Coveralls config.

Since you are using the Coveralls GitHub Action, the tokens attached to your upload submissions are "special" in the sense that, instead of needing to be the traditional Coveralls Repo Token, they can be the GitHub App token from our Action. And as long as you are not explicitly blocking PRs from forks from leveraging our Action's token, then PRs from external collaborators should be treated like any other on your project.

Other access differences that could cause issues or prevent successful builds:

The only thing I did notice that could be causing periodic issues---and more with the Web UI than with uploads to our API---is that this Coveralls Repo does not currently have what we refer to as a "repo owner." I will leave out the explanation, and just say that this can cause issues receiving GitHub Status Updates---and sometimes PR Comments as well---but it wouldn't typically cause a 500 Internal Server Error from the API.

But to remove this as a factor, nevertheless, I gave the repo an owner.

Interestingly, I tried to make you the owner since I know you are working on the repo, but I couldn't find a Coveralls user account for your GitHub username being used here. I assume this means you are using Coveralls without logging in, which is possible for public / open source repos.

None of the other users contributing to this recent PR you mentioned had Coveralls user accounts either, so, in order to make someone the owner at Coveralls, I made this user owner, since they had an active Coveralls session and seemed to belong to many repos from the collective org.

My confusion: I am somewhat confused about your surprise that this build succeeded. I assume that's because we have different suppositions about the requirements for a successful coverage report submission to Coveralls, which is why I tried to clear up any confusion over factors of access such as that of a maintainer vs. an external contributor. (They should be the same.)

That said, I might be missing something about why you expected this build to fail---so, if you think so, please let me know.

For the meantime, I believe we can factor out the committer of any commits being built for this project in CI, and expect them to succeed.

As a result, I'd like to know if any patterns do emerge if you continue receiving 500 errors from the Coveralls API for any of your uploads.

The fact that this build succeeded: https://coveralls.io/builds/68499778

Does at least tell me that each of your constituent uploads has succeeded at one time, which is helpful.

For us, a 500 error is typically an infrequent "one-off" type of error, at least if coverage report submissions have ever been successful and nothing has changed with your configuration.

I am somewhat hopeful that giving your repo an owner at Coveralls removes the possibility of that being a factor, which could relate to something I'm overlooking about its role in submissions---like, perhaps there's something token-related in one of the later stages of a build that I'm overlooking that could cause a call to GitHub to hang, leading to a timeout---a 504 or 502---that's being improperly handled and misreported as a 500.

Please let me know the next time you get a 500 error.

Alternatively, I'd love to know if the problem clears up after this.

niccokunzmann commented 1 week ago

Thanks for taking so much time! For now, we had this PR and all seems fine again: https://github.com/collective/icalendar/pull/694 If you are alright with it, we can close this. When the errer occurs again, we can link to this.

So, it looks like coverage reports were sent once for the build, originally, on Jul 07, then sent again, on Jul 09.

The contributor started one failing. When I restarted the same, coverage submission worked then.

afinetooth commented 1 week ago

My pleasure, @niccokunzmann.

On that build, got it. Thanks for the update.

And yes, I'll close this and we can reference it or reopen if it happens again.

Take care.

coverallsapp / github-action

⚠️ Internal server error. Please contact Coveralls team. / report submission fails for external contributors #217