Coveralls stopped working with no code changes in project

superm1 commented 2 months ago

Recently fwupd CI started to break on any jobs that upload to coveralls.

We're using coverallsapp/github-action@3dfc5567390f6fa9267c0ee9c251e4c8c3f18949 and here is the failure in CI logs.

coveralls report        
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    COVERALLS_DEBUG: false
    COVERALLS_CARRYFORWARD_FLAGS: 
    COVERALLS_FLAG_NAME: run-debian-x86_64
    COVERALLS_PARALLEL: true
    COVERALLS_ENDPOINT: https://coveralls.io/
    COVERALLS_GIT_BRANCH: 
    COVERALLS_GIT_COMMIT: 
    COVERALLS_REPO_TOKEN: ***
    COVERALLS_COMPARE_REF: 
    COVERALLS_COMPARE_SHA: 
    COVERALLS_SOURCE_HEADER: github-action

⠀⠀⠀⠀⠀⠀⣿
⠀⠀⠀⠀⠀⣼⣿⣧⠀⠀⠀⠀⠀⠀⠀ ⣠⣶⣾⣿⡇⢀⣴⣾⣿⣷⣆ ⣿⣿⠀⣰⣿⡟⢸⣿⣿⣿⡇ ⣿⣿⣿⣷⣦⠀⠀⢠⣿⣿⣿⠀⠀⣿⣿⠁⠀⣼⣿⡇⠀⢀⣴⣾⣿⡷
⠶⣶⣶⣶⣾⣿⣿⣿⣷⣶⣶⣶⠶  ⣸⣿⡟ ⠀⢠⣿⣿⠃⠈⣿⣿⠀⣿⣿⢠⣿⡿⠀⣿⣿⣧⣤⠀⢸⣿⡇⣠⣿⡿⠀⢠⣿⡟⣿⣿⠀⢸⣿⡿⠀⠀⣿⣿⠃⠀⢸⣿⣧⣄
⠀⠀⠙⢻⣿⣿⣿⣿⣿⡟⠋⠁⠀⠀ ⣿⣿⡇⠀ ⢸⣿⣿⠀⣸⣿⡟⠀⣿⣿⣾⡿⠁ ⣿⣿⠛⠛⠀⣿⣿⢿⣿⣏⠀⢀⣿⣿⣁⣿⣿⠀⣾⣿⡇⠀⢸⣿⡿⠀⠀⡀⠙⣿⣿⡆
⠀⠀⢠⣿⣿⣿⠿⣿⣿⣿⡄⠀⠀⠀ ⠙⢿⣿⣿⠇⠈⠿⣿⣿⡿⠋⠀⠀⢿⣿⡿⠁⠀⢸⣿⣿⣿⡇⢸⣿⣿⠀⣿⣿⣄⣾⣿⠛⠛⣿⣿⢠⣿⣿⣿ ⣼⣿⣿⣿ ⣿⣿⡿⠋⠀
⠀⢀⣾⠟⠋⠀⠀⠀⠙⠻⣷⡀⠀⠀

  v0.6.10

🔍 Detected coverage file: coverage.xml
⚠️ Coverage reporter does not yet know how to process this file: coverage.xml
🚨 Nothing to report

Here is an example failure: https://github.com/fwupd/fwupd/actions/runs/8877359982/job/24370866636?pr=7179

Here is one that passed (again no changes on our side for coveralls).

coveralls report        
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    COVERALLS_DEBUG: false
    COVERALLS_CARRYFORWARD_FLAGS: 
    COVERALLS_FLAG_NAME: run-debian-x86_64
    COVERALLS_PARALLEL: true
    COVERALLS_ENDPOINT: https://coveralls.io
    COVERALLS_GIT_BRANCH: 
    COVERALLS_GIT_COMMIT: 
    COVERALLS_REPO_TOKEN: ***
    COVERALLS_COMPARE_REF: 
    COVERALLS_COMPARE_SHA: 
    COVERALLS_SOURCE_HEADER: github-action

⠀⠀⠀⠀⠀⠀⣿
⠀⠀⠀⠀⠀⣼⣿⣧⠀⠀⠀⠀⠀⠀⠀ ⣠⣶⣾⣿⡇⢀⣴⣾⣿⣷⣆ ⣿⣿⠀⣰⣿⡟⢸⣿⣿⣿⡇ ⣿⣿⣿⣷⣦⠀⠀⢠⣿⣿⣿⠀⠀⣿⣿⠁⠀⣼⣿⡇⠀⢀⣴⣾⣿⡷
⠶⣶⣶⣶⣾⣿⣿⣿⣷⣶⣶⣶⠶  ⣸⣿⡟ ⠀⢠⣿⣿⠃⠈⣿⣿⠀⣿⣿⢠⣿⡿⠀⣿⣿⣧⣤⠀⢸⣿⡇⣠⣿⡿⠀⢠⣿⡟⣿⣿⠀⢸⣿⡿⠀⠀⣿⣿⠃⠀⢸⣿⣧⣄
⠀⠀⠙⢻⣿⣿⣿⣿⣿⡟⠋⠁⠀⠀ ⣿⣿⡇⠀ ⢸⣿⣿⠀⣸⣿⡟⠀⣿⣿⣾⡿⠁ ⣿⣿⠛⠛⠀⣿⣿⢿⣿⣏⠀⢀⣿⣿⣁⣿⣿⠀⣾⣿⡇⠀⢸⣿⡿⠀⠀⡀⠙⣿⣿⡆
⠀⠀⢠⣿⣿⣿⠿⣿⣿⣿⡄⠀⠀⠀ ⠙⢿⣿⣿⠇⠈⠿⣿⣿⡿⠋⠀⠀⢿⣿⡿⠁⠀⢸⣿⣿⣿⡇⢸⣿⣿⠀⣿⣿⣄⣾⣿⠛⠛⣿⣿⢠⣿⣿⣿ ⣼⣿⣿⣿ ⣿⣿⡿⠋⠀
⠀⢀⣾⠟⠋⠀⠀⠀⠙⠻⣷⡀⠀⠀

  v0.6.10

🔍 Detected coverage file: coverage.xml
⭐️ Running in parallel mode. You must call the webhook after all jobs finish: `coveralls done --build-number 88683382[29](https://github.com/fwupd/fwupd/actions/runs/8868338229/job/24347822383#step:8:31)`
  ·job_flag: run-debian-x86_64
🚀 Posting coverage data to https://coveralls.io//api/v1/jobs
---
✅ API Response: {"message":"Coverage for parallel build uploaded","url":"https://coveralls.io/builds/67177375"}
- 💛, Coveralls

Example: https://github.com/fwupd/fwupd/actions/runs/8868338229/job/24347822383

CC @hughsie

superm1 commented 2 months ago

Maybe the same problem as https://github.com/coverallsapp/github-action/issues/205 ? The confusing thing to me though is that both the pass and fail used 0.6.10 it looks like.

superm1 commented 2 months ago

Double checked with --debug on the job added, and calling pip install coverage before running it doesn't change anything.

Run coveralls report --debug       
  coveralls report --debug       
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    COVERALLS_DEBUG: true
    COVERALLS_CARRYFORWARD_FLAGS: 
    COVERALLS_FLAG_NAME: run-debian-x86_64
    COVERALLS_PARALLEL: true
    COVERALLS_ENDPOINT: https://coveralls.io
    COVERALLS_GIT_BRANCH: 
    COVERALLS_GIT_COMMIT: 
    COVERALLS_REPO_TOKEN: ***
    COVERALLS_COMPARE_REF: 
    COVERALLS_COMPARE_SHA: 
    COVERALLS_SOURCE_HEADER: github-action

⠀⠀⠀⠀⠀⠀⣿
⠀⠀⠀⠀⠀⣼⣿⣧⠀⠀⠀⠀⠀⠀⠀ ⣠⣶⣾⣿⡇⢀⣴⣾⣿⣷⣆ ⣿⣿⠀⣰⣿⡟⢸⣿⣿⣿⡇ ⣿⣿⣿⣷⣦⠀⠀⢠⣿⣿⣿⠀⠀⣿⣿⠁⠀⣼⣿⡇⠀⢀⣴⣾⣿⡷
⠶⣶⣶⣶⣾⣿⣿⣿⣷⣶⣶⣶⠶  ⣸⣿⡟ ⠀⢠⣿⣿⠃⠈⣿⣿⠀⣿⣿⢠⣿⡿⠀⣿⣿⣧⣤⠀⢸⣿⡇⣠⣿⡿⠀⢠⣿⡟⣿⣿⠀⢸⣿⡿⠀⠀⣿⣿⠃⠀⢸⣿⣧⣄
⠀⠀⠙⢻⣿⣿⣿⣿⣿⡟⠋⠁⠀⠀ ⣿⣿⡇⠀ ⢸⣿⣿⠀⣸⣿⡟⠀⣿⣿⣾⡿⠁ ⣿⣿⠛⠛⠀⣿⣿⢿⣿⣏⠀⢀⣿⣿⣁⣿⣿⠀⣾⣿⡇⠀⢸⣿⡿⠀⠀⡀⠙⣿⣿⡆
⠀⠀⢠⣿⣿⣿⠿⣿⣿⣿⡄⠀⠀⠀ ⠙⢿⣿⣿⠇⠈⠿⣿⣿⡿⠋⠀⠀⢿⣿⡿⠁⠀⢸⣿⣿⣿⡇⢸⣿⣿⠀⣿⣿⣄⣾⣿⠛⠛⣿⣿⢠⣿⣿⣿ ⣼⣿⣿⣿ ⣿⣿⡿⠋⠀
⠀⢀⣾⠟⠋⠀⠀⠀⠙⠻⣷⡀⠀⠀

  v0.6.10

🔍 Detected coverage file: coverage.xml
⚠️ Coverage reporter does not yet know how to process this file: coverage.xml
⏱️ Report parsing: 00:00:00.9749[28](https://github.com/fwupd/fwupd/actions/runs/8881167836/job/24383026409#step:9:30)631
🚨 Nothing to report

superm1 commented 2 months ago

Something must have changed in the container we use. I changed our job to upload artifacts including the coverage file and I see it it's missing data. Sorry for the noise!

afinetooth commented 2 months ago

Hi @superm1. No worries. Based on that error from our API, it means our integration (Coverage Reporter) sees a file, coverage.xml, in your environment but doesn't recognize the format and can't parse it.

If the format you're using is one of the formats we support, then this can be fixed by using the --format flag (or the format: input option if you're using the Coveralls GitHub Action), to specify the format of the coverage report(s) you want to upload.

For instance, given you're in python, if you've converted your .coverage file into XML format with the coverage xml command, then the specific XML format of your new coverage report will be Cobertura XML, and therefore you should use --format cobertura.

Using --format is not required in general, but is recommended since it will tell our integration (Coverage Reporter) that it should only focus on reports of that format and should ignore reports in other formats, which will prevent errors when you have duplicate reports in different formats in the same environment.

coverallsapp / github-action

Coveralls stopped working with no code changes in project #207