lemurheavy / coveralls-public

The public issue tracker for coveralls.io
http://coveralls.io
124 stars 7 forks source link

Can I use coveralls from a single repo with two different code bases? #1640

Closed dominikwilkowski closed 2 years ago

dominikwilkowski commented 2 years ago

I have a repo that includes a node and a rust implementation of the same thing. I have coveralls setup for the node implementation but now I don't know how to add the rust one. I have a lcov.info file for the rust coverage but since it comes from the same repo I fear it would just override the node one...

Is there a way to create multiple coveralls project for the same repo basically?

afinetooth commented 2 years ago

Hi, @dominikwilkowski.

While there's no way to create multiple coveralls projects for a single github repo (the relationship is 1-to-1), yours is a common case and this is the recommended approach:

You will probably want to configure coveralls.io to receive parallel builds for your project.

Specifically, what you'll do is modify your CI config file to build your project in parts, in parallel, often called a matrix build, where you'll send the node project coverage to coveralls.io in one parallel "job" and send the rust project coverage in another parallel job.

You're ahead of the game with a lcov.info file for your rust implementation. Nearly all coveralls integrations accept and process LCOV format coverage reports.

One integration that will be particularly useful / efficient for your use case is the Coveralls Github Action, which requires LCOV reports.

This usage example from the README demonstrates how to configure your Github Actions config YAML to send parallel jobs to coveralls.io.

In addition, the coveralls-node-demo project demonstrates the approach to configuring parallel builds for coveralls.io in the more general case, providing examples for Github Actions, CircleCI and Travis.

A so-called "matrix build" might not make sense for your project as they typically apply to running the same codebase against multiple different environments (like OS's).

Instead, you might just want to set up your project's CI builds with two (2) parallel jobs, in three (3) steps, like this:

Example using Coveralls Github Action

  1. Step 1: Run the tests for the node portion of your project. Convert the coverage report to LCOV format if necessary. Send the coverage report to coveralls with:
jobs:
  tests:
    steps:
      name: Coveralls Node Parallel
      uses: coverallsapp/github-action@master
      with:
        github-token: ${{ secrets.github_token }}
        parallel: true
        path-to-lcov: <path to `lcov.info` if not at `./coverage/lcov.info`>
        flag-name: node-job
  1. Step 2: Run the tests for the rust portion of your project. Convert the coverage report to LCOV format if necessary (sounds like it's not necessary). Send the coverage report to coveralls with:
jobs:
  tests:
    steps:
      [...]

      name: Coveralls Rust Parallel
      uses: coverallsapp/github-action@master
      with:
        github-token: ${{ secrets.github_token }}
        parallel: true
        path-to-lcov: <path to `lcov.info` if not at `./coverage/lcov.info`>
        flag-name: rust-job
  1. Step 3: Close the parallel build by sending the parallel build (closed) webhook, like this:
  finish:
    needs: tests
    steps:
      - name: Coveralls Parallel Build Finished
        uses: coverallsapp/github-action@v1.1.0
        with:
          github-token: ${{ secrets.github_token }}
          parallel-finished: true

The third step here is required. You must tell coveralls.io to close the parallel build before it will merge coverage reports from each job and calculate an aggregate coverage % for the full project.

Then you'll have both a single coverage report for the project as a whole, as well as individual coverage reports for each of the constituent jobs.

Hope that helps!

dominikwilkowski commented 2 years ago

Thank you for the detailed answer @afinetooth.

This has worked really well and for others you can see the PR here: https://github.com/dominikwilkowski/cfonts/pull/51

Note some things I had to do with base-path but works like a charm now :)

dominikwilkowski commented 2 years ago

Actually I'm confused now. Why is it now showing only rust? https://coveralls.io/github/dominikwilkowski/cfonts

The pull request (https://coveralls.io/builds/49785114) itself seems to show the both languages but once merged it now only shows rust or am I reading this wrong somehow?

afinetooth commented 2 years ago

Hi @dominikwilkowski.

So it looks like you're only configured for parallel builds on pull_requests. For example, for this PR build: https://coveralls.io/builds/49785114

You can see your two parallel jobs: Screen Shot 2022-06-07 at 9 48 16 AM

And your SOURCE FILES table shows both rust and JS files: Screen Shot 2022-06-07 at 9 48 46 AM

On the other hand, your push type builds on your default branch (released), still appear to be non-parallel. Example: https://coveralls.io/builds/49785123

Only one job: Screen Shot 2022-06-07 at 9 50 05 AM

Only rust files: Screen Shot 2022-06-07 at 9 50 32 AM

The reason you're seeing the same thing (only rust files in your SOURCE FILES table) on your main repo page: https://coveralls.io/github/dominikwilkowski/cfonts

Is because the repo page always reflects the last build on your default branch (released).

That's because it is meant to reflect the coverage % for the production (or production-ready) version of your code. Same as your badge coverage is meant to represent.

If you will share your config file, we can see why your push builds aren't building in parallel.

My guess, right now, since the one job in your push builds reads, .2: https://coveralls.io/builds/49785123

Screen Shot 2022-06-07 at 9 53 18 AM

Is that parallel: true is missing for one step and the second job (rust) is replacing the first job (JS).

Have a look at the usage example for parallel builds with the Coveralls Github Action again: https://github.com/marketplace/actions/coveralls-github-action#complete-parallel-job-example

Note how the on: key specifies the same build behavior for both push and pull_request type builds:

on: ["push", "pull_request"]

name: Test Coveralls Parallel
[...]
dominikwilkowski commented 2 years ago

You can see my config here: https://github.com/dominikwilkowski/cfonts/blob/released/.github/workflows/testing.yml

on:
  push:
    branches:
      - 'released'
    tags:
      - '!v*'
  pull_request:
    branches:
      - 'released'

And I have parallel: true on both jobs. The only difference is that I have to specify base-path: ./nodejs on one of them presumably because the lcov.info paths perhaps?

(for the chance it changes later here the relevant bits)

# -------------------------------| COVERAGE |------------------------------- #
  # ***********
  # COVERAGE - nodejs
  # ***********
  coverage-nodejs:
    needs: test-nodejs
    strategy:
      matrix:
        node: [16]
        os:
          - ubuntu-latest
    runs-on: ${{ matrix.os }}
    defaults:
      run:
        working-directory: ./nodejs

    steps:
      - uses: actions/checkout@v2

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node }}

      - name: Get yarn cache
        id: yarn-cache
        run: echo "::set-output name=dir::$(yarn cache dir)"

      - name: Cache dependencies
        uses: actions/cache@v2
        with:
          path: ${{ steps.yarn-cache.outputs.dir }}
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-
      - name: Yarn install dependencies
        run: yarn install --frozen-lockfile

      - name: Build files
        run: yarn build

      - name: Produce Coverage
        run: yarn jest --coverage

      - name: Print last few lines of lcov.info
        run: tail ./coverage/lcov.info

      - name: Upload to coveralls
        uses: coverallsapp/github-action@master
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          path-to-lcov: ./nodejs/coverage/lcov.info
          parallel: true
          base-path: ./nodejs

  # ***********
  # COVERAGE - rust
  # ***********
  coverage-rust:
    needs: test-rust
    strategy:
      matrix:
        os:
          - ubuntu-latest
    runs-on: ${{ matrix.os }}
    defaults:
      run:
        working-directory: ./rust

    steps:
      - uses: actions/checkout@v2

      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable

      - name: Run Makefile
        run: make

      - name: Install Rust
        run: rustup toolchain install stable --component llvm-tools-preview

      - name: Install cargo-llvm-cov
        uses: taiki-e/install-action@cargo-llvm-cov

      - name: Generate code coverage
        run: cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info

      - name: Print last few lines of lcov.info
        run: tail ./lcov.info

      - name: Upload to coveralls
        uses: coverallsapp/github-action@master
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          path-to-lcov: ./rust/lcov.info
          parallel: true

  # ***********
  # COVERAGE - finalizing
  # ***********
  coverage-finish:
    needs: [coverage-nodejs, coverage-rust]
    strategy:
      matrix:
        os:
          - ubuntu-latest
    runs-on: ${{ matrix.os }}
    steps:
      - name: Coveralls Parallel Build Finished
        uses: coverallsapp/github-action@v1.1.0
        with:
          github-token: ${{ secrets.github_token }}
          parallel-finished: true
afinetooth commented 2 years ago

Hi @dominikwilkowski. Ok, yes, I don't see anything in your config that indicates why we'd only be getting non-parallel builds from released.

I cover some observed details on that below, but the important thing is the next step, which is to enable verbose mode on your coveralls integration so we can search for clues in the debug info that's added to your CI build logs.


Next step: Enable verbose mode for the Coveralls Github Action:


Details from CI build logs: Relevant, if not conclusive:

Neither job shows any failures, but the second job appears to have replaced the first: https://coveralls.io/builds/49785123

Screen Shot 2022-06-08 at 10 21 31 AM

Again, verbose / debug info should help with clues why.

dominikwilkowski commented 2 years ago

I enabled the NODE_COVERALLS_DEBUG env var and re-ran the action here: https://github.com/dominikwilkowski/cfonts/actions/runs/2452417481

Also just in case the repeat of the action doesn't actually take the new env var into account I created a new PR here: https://github.com/dominikwilkowski/cfonts/pull/52

And the PR shows again both builds: https://coveralls.io/builds/49859798 I assume it will drop back to only rust once merged just like before since we didn't change anything material.

afinetooth commented 2 years ago

Thanks, @dominikwilkowski.

I'm not seeing the debug output I'd expect from you adding the NODE_COVERALLS_DEBUG=1 env var. I could not find NODE_COVERALLS_DEBUG=1 in the .github/workflows/testing.yml file for that commit, so maybe you added it as a repo secret?

In any case, to benefit from that debug info, it would be great if you could try adding that again. It should be as easy as adding and env: key to each coveralls-related step, something like:

name: Upload to coveralls
        uses: coverallsapp/github-action@master
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          path-to-lcov: ./nodejs/coverage/lcov.info
          parallel: true
          base-path: ./nodejs
        env:
          NODE_COVERALLS_DEBUG=1

Aside from that, like you do I more or less expect to see the same dropping of one job in the next push build on released. If only because, at this point, we haven't really changed anything.

But I do believe there will be clues in the verbose logs for both the next PR build and the next push build.

dominikwilkowski commented 2 years ago

Hey @afinetooth

I just added it to the testing yaml file as well. I did add it as a secret. Hope this works now

afinetooth commented 2 years ago

HI @dominikwilkowski. Me too!. Let me know.

dominikwilkowski commented 2 years ago

@afinetooth So do you want me to merge it? I will do that now and see. I can always open a new PR up for debugging.

Thank you btw for your excellent help. It's been a great experience.

dominikwilkowski commented 2 years ago

I actually merged another PR yesterday https://github.com/dominikwilkowski/cfonts/pull/53 that also resulted in the right view on the PR in coveralls so it showed both rust and nodejs (https://coveralls.io/builds/49913169) but upon merging now only shows rust alone (https://coveralls.io/github/dominikwilkowski/cfonts). Any help is appreciated.

dominikwilkowski commented 2 years ago

Ok that PR is now also merged (interesting that the finishing job doesn't really generate verbose output like the other once https://github.com/dominikwilkowski/cfonts/runs/6840306155?check_suite_focus=true) and coveralls still only shows rust as a language.

afinetooth commented 2 years ago

Hi, @dominikwilkowski.

Thanks for the updates.

This behavior is disconcerting since I don't see any reason for it in your config file.

That's why I need to dig deeper into your verbose verbose CI build logs.

However, using the last PR build and base build pairing you mention as examples, I'm not seeing verbose output in those build logs:

Builds

PR build: https://coveralls.io/builds/49913169

Screen Shot 2022-06-13 at 11 14 11 AM

Base build: https://coveralls.io/builds/49785123

Screen Shot 2022-06-13 at 11 14 35 AM

Logs

PR build log (Job 1): https://github.com/dominikwilkowski/cfonts/runs/6826526223?check_suite_focus=true

Screen Shot 2022-06-13 at 11 16 09 AM

PR build log (Job 2): https://github.com/dominikwilkowski/cfonts/runs/6826566931?check_suite_focus=true

Screen Shot 2022-06-13 at 11 16 38 AM

And:

Base build log (First job - missing from Coveralls build page): https://github.com/dominikwilkowski/cfonts/runs/6801302087?check_suite_focus=true

Screen Shot 2022-06-13 at 11 18 48 AM

Base build log (Second job - Appears in Coveralls build page as Job .4): https://github.com/dominikwilkowski/cfonts/runs/6801317417?check_suite_focus=true

Screen Shot 2022-06-13 at 11 19 47 AM

_BTW, you're right, the "Coveralls Finish" job (aka. "Parallel build closed webhook" job): https://github.com/dominikwilkowski/cfonts/runs/6801345192?check_suite_focus=true_

Does not currently have any helpful debug output for this integration:

Screen Shot 2022-06-13 at 11 23 37 AM

I'm hoping we'll add some soon, but for the success case, there's not all that much to see beyond the BUILD ID in the original request, and, unless there's a failure, all that happens in the success case is that the build's status = done, meaning it's closed at coveralls.io, which we can easily confirm in the web UI. That's how I use it.

The meat of the sandwich is in the two coverage job POSTs, particularly the JSON object representing the contents of the JSON file sent as a form file upload with each POST.

ANYWAY: I'm not seeing any debug output in the CI logs, as you can see from the screenshots above), and I'm not finding the NODE_COVERALLS_DEBUG=1 env var in your config file:

Screen Shot 2022-06-13 at 11 28 12 AM

So, questions:

  1. Did you add NODE_COVERALLS_DEBUG=1 as a secret instead?
  2. Are you seeing debug output somewhere? Am I just missing it?

If you did run some verbose builds, then destroy / replace them due to security concerns with having them up at Github, can you re-run the builds, export the verbose logs and share those with me at support@coveralls.io? If you mention this issue, I'll get them.


dominikwilkowski commented 2 years ago

Hey, Let's pick the last PR that I have merged. I'm also happy to open a new one for you to investigate this. Do let me know if that would help.

Last PR: https://github.com/dominikwilkowski/cfonts/pull/55 Actions for it: https://github.com/dominikwilkowski/cfonts/actions/runs/2479518322

I hope this helps you look into this

To your questions:

Did you add NODE_COVERALLS_DEBUG=1 as a secret instead?

I ended up doing both. The current workflow actually includes them on the default branch here: https://github.com/dominikwilkowski/cfonts/blob/released/.github/workflows/testing.yml

Are you seeing debug output somewhere? Am I just missing it?

Yes I am seeing the debug output for the parallel builds. Screenshots below (for the links above)

Nodejs:

image

Rust:

image
afinetooth commented 2 years ago

Ok, that's great. That has everything I need.

So, as discussed, the PR builds are acting normally. Everything I see in those logs is expected. And the open question is why the push builds aren't acting the same.

In this context, the base build / push build is this one: https://coveralls.io/builds/49940082

And luckily we have verbose logs for both of its jobs, here (nodejs): https://github.com/dominikwilkowski/cfonts/runs/6842439021?check_suite_focus=true

Screen Shot 2022-06-14 at 11 36 35 AM

And here (rust): https://github.com/dominikwilkowski/cfonts/runs/6842444372?check_suite_focus=true

Screen Shot 2022-06-14 at 11 36 55 AM

Notice that both show 200 success messages, and display the resulting new job URL.

The difference is that the first job (nodejs) is missing from coveralls.io: https://coveralls.io/jobs/101054720

Screen Shot 2022-06-14 at 11 33 22 AM

While the second job (rust) is there: https://coveralls.io/jobs/101054767

Screen Shot 2022-06-14 at 11 34 33 AM


So it's as though the second job (rust) replaced the first (nodejs).

For which there would be two (2) possible reasons that I'm aware of:

  1. The build doesn't know it's parallel
  2. Both jobs have identical information, making the second job appear to be a duplicate of the first job.

But I don't see evidence for either of those two reasons in the verbose build logs.

So, unfortunately, I need to keep searching for some other, unexpected reason, which I'll do and get back here, asap.

afinetooth commented 2 years ago

@dominikwilkowski I think I've got it.

Please try adding a flag-name parameter to the config for each coveralls/upload job, per the instructions here: https://github.com/marketplace/actions/coveralls-github-action#inputs

While, under Inputs section there, the flag-name option is listed as optional, it's actually required in the case of a parallel job (parallel: true):

parallel optional Set to true for parallel (or matrix) based steps, where multiple posts to Coveralls will be performed in the check. flag-name needs to be set and unique, e.g. flag-name: run-${{ matrix.test_number }}

Visually:

Screen Shot 2022-06-14 at 11 44 54 AM

Since your jobs are not in a matrix, you don't have to use a variable in the name, just use any name that's unique between the two jobs.

Let us know how it goes.

dominikwilkowski commented 2 years ago

This worked! yay! Thank you so much for all your help. amazing experience.

afinetooth commented 2 years ago

You're welcome!