Is there a way to post coverage from Github Actions in parallel mode?

miguelnietoa commented 1 year ago

Hi there!

Something like mix coveralls.github --parallel true and then setup another job sending the webhook to https://coveralls.io/webhook?repo_token=…

I know that mix coveralls.post does allow this parallel option.

shamshirz commented 5 months ago

Hey, it's unlikely you are experiencing this problem still, but I was!

Problem

I want to run my partitioned test suite and post a coverage report of the merged partitions. The mix coveralls.github isn't working and isn't giving me any clues!

Solution

Combination of mix coveralls.lcov (read: Changed to a different report format) + the coveralls github action.

This psuedo-sample github action below shows how it worked for us.

For each Test Partition
- Build the coverage report for your partition
- Upload it with the action
After, use the action to post that the jobs are complete

test:
    needs: build
    runs-on: ubuntu-latest
    name: Test-OTP ${{matrix.otp}}/Elixir ${{matrix.elixir}}/${{matrix.test_partition}}
    strategy:
      matrix:
        test_partition: [1, 2]
        otp: [26.1.1]
        elixir: [1.15.6]
    services:
      db: …
    steps:
    - name: Install OTP and Elixir
    - other setup like cache etc…
    - name: Run Tests and Generate Coverage Report
      run: |
        MIX_TEST_PARTITION=${{ matrix.test_partition }} mix coveralls.lcov --partitions 2
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    - name: Upload Parallel Coverage Report
      uses: coverallsapp/github-action@v2
      with:
        flag-name: run-${{matrix.test_partition}}
        parallel: true

  finish:
    needs: test
    if: ${{ always() }}
    runs-on: ubuntu-latest
    steps:
      - name: Close parallel build
        uses: coverallsapp/github-action@v2
        with:
          parallel-finished: true
          carryforward: "run-1,run-2"

Fun keywords that may help others find this:

✅ API Response: {"error":"No build matching CI build number 8453443938 found"}

miguelnietoa commented 5 months ago

@shamshirz thanks a lot!

When I initially tackled this issue, I constructed the request to Coveralls in the following way: https://github.com/kommitters/stellar_sdk/blob/d0ad69c4cb021c0da7afb776403d7fb7100effd7/.github/workflows/ci.yml

Would you mind sharing your thoughts on whether your suggested approach might be more effective than mine?

shamshirz commented 4 months ago

tl;dr I think yours is equally good 👍 No complaint

Mine: uses: coverallsapp/github-action@v2 for pushing partial coverage and for notifying parallel is done Yours: mix coveralls.github for partial coverage and curl directly for notifying parallel is done

I don't think it makes much difference. If it could've gotten the mix task to work for both, then I likely would've thought that was the best of both worlds. Additionally in case you run into this, I found that the overhead of running mix test --cover (equivalent to mix coveralls.X) was really high! Whether you do a portion of tests or the whole suite, it has to compile all of the files again within the task.

Ultimately, I ended up going back to a non-parallel approach and we only update the coverage badge after merges into main, rather than doing the coverage check as part of each PR.

parroty / excoveralls