search

infertux / bashcov

Code coverage tool for Bash
MIT License
148 stars 19 forks source link

Coverage % changes each time. #82

Open Minituff opened 10 months ago

Minituff commented 10 months ago

Hello,

I'm new to using Ruby so I apologized if this is a simple question, but every time I run my tests my coverage\index.html is giving me a different output.

The strange thing is, the output is correct and consistent within my Github Actions pipeline.

Im using: bashcov 3.1.1 with Bash 5.0, Ruby 3.3.0, and SimpleCov 0.21.2

My `.simplecov` file ```ruby require 'simplecov' require 'simplecov-cobertura' require "simplecov-html" # frozen_string_literal: true # SimpleCov.formatter = SimpleCov::Formatter::CoberturaFormatter # Converts the `.resultset.json` to `coverage.xml` # SimpleCov::Formatter::JSONFormatter, # This formatter breaks the build # Use multiple formatters SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([ SimpleCov::Formatter::SimpleFormatter, SimpleCov::Formatter::CoberturaFormatter, SimpleCov::Formatter::HTMLFormatter, ]) # .simplecov SimpleCov.start 'rails' do command_name 'Unit Tests' enable_coverage :branch primary_coverage :branch add_filter %r{^/snippets/} add_filter %r{^/.git/} add_filter %r{^/tests/} add_filter "pkg/test.sh" add_group "Pkg scripts", "/pkg" enable_coverage_for_eval # Must be at the bottom and Must be here, even though it throws a 'command not found' error end ```
Here is the relevant part of my Gitlab CI workflow ```yml - uses: ruby/setup-ruby@v1 with: ruby-version: head - name: Install Ruby dependencies run: | gem install bashcov simplecov-cobertura simplecov-html - name: Run unit tests run: bashcov ./tests/tests.sh && ls -la coverage - name: Upload reports to Codecov uses: codecov/codecov-action@v4-beta # uses: codecov/codecov-action@v3 env: # Required for v4 CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} with: fail_ci_if_error: true files: coverage/coverage.xml # The json file will not work here ```

This is the output when I run the tests locally changes each time, but ranges from 50-65%

image

This is the output generated from my testing pipeline. It's always accurate and consistent (~90%)

image

Are you able to help me determine why the coverage is not the same when run codecov locally?

How to replicate the issue

  1. Clone my repo git clone git@github.com:Minituff/nautical-backup.git
  2. Install dependencies: gem install bashcov simplecov-cobertura simplecov-html
  3. Run the tests bashcov ./tests/tests.sh && ls -la coverage

My repo is open source so feel free to look around if I did not include something https://github.com/Minituff/nautical-backup

Thanks!

tomeon commented 10 months ago

Hello @Minituff -- I've looked into this and I'm unable to reproduce the issue. Here's the branch of my nautical-backup fork that I'm using to investigate your issue. I've had to make a few changes to nautical-backup in order to test on my system, and I haven't replicated other aspects of your environment precisely -- e.g., I am using Bash 5.2.15 rather than 5.0 and Ruby 3.2 rather than 3.3 -- so this may invalidate my "cannot reproduce". However, I encourage you to try using my branch and see if the issue persists.

Test results:

$ for i in {1..10}; do printf 1>&2 -- '# %s\n' "$i"; \rm -rf ./coverage; bundle exec bashcov --bash-path "$(type -P bash)" ./tests/tests.sh |& grep '^Coverage report generated'; done
# 1
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
# 2
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
# 3
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
# 4
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
# 5
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
# 6
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
# 7
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
# 8
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
# 9
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
# 10
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage/coverage.xml. 960 / 1407 LOC (68.23%) covered
Coverage report generated for /run/current-system/sw/bin/bash ./tests/tests.sh to /home/matt/src/github.com/Minituff/nautical-backup/coverage. 960 / 1407 LOC (68.23%) covered.
Minituff commented 10 months ago

Hey @tomeon, thank you for putting in so much work to help me out with this.

I think I may have figured it out.

I also ran a few tests similar to your and here are my results: ```console ## My main branch (WSL2) Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage/coverage.xml. 254 / 407 LOC (62.41%) covered; 0 / 0 BC (100.00%) covered Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage. 254 / 407 LOC (62.41%) covered. Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage/coverage.xml. 254 / 405 LOC (62.72%) covered; 0 / 0 BC (100.00%) covered Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage. 254 / 405 LOC (62.72%) covered. Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage/coverage.xml. 250 / 401 LOC (62.34%) covered; 0 / 0 BC (100.00%) covered Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage. 250 / 401 LOC (62.34%) covered. ## Then I added your .simplecov file to my main branch (WSL2) Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage/coverage.xml. 579 / 1726 LOC (33.55%) covered Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage. 579 / 1726 LOC (33.55%) covered. Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage/coverage.xml. 608 / 1747 LOC (34.80%) covered Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage. 608 / 1747 LOC (34.8%) covered. Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage/coverage.xml. 638 / 1759 LOC (36.27%) covered Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage. 638 / 1759 LOC (36.27%) covered. ## Your testing branch (WSL2) Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage/coverage.xml. 545 / 1451 LOC (37.56%) covered Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage. 545 / 1451 LOC (37.56%) covered. Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage/coverage.xml. 585 / 1457 LOC (40.15%) covered Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage. 585 / 1457 LOC (40.15%) covered. Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage/coverage.xml. 463 / 1426 LOC (32.47%) covered Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup-1/coverage. 463 / 1426 LOC (32.47%) covered. ```

I am using WSL2 (Ubuntu 20.04) on my Windows 11 PC so I tried booting up a Ubuntu VM.

Here are the results from my Ubuntu VM: ```console ## My main branch (Ubuntu VM) Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage/coverage.xml. 379 / 411 LOC (92.21%) covered; 0 / 0 BC (100.00%) covered Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage. 379 / 411 LOC (92.21%) covered. Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage/coverage.xml. 379 / 411 LOC (92.21%) covered; 0 / 0 BC (100.00%) covered Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage. 379 / 411 LOC (92.21%) covered. Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage/coverage.xml. 379 / 411 LOC (92.21%) covered; 0 / 0 BC (100.00%) covered Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage. 379 / 411 LOC (92.21%) covered. ## Then I added your .simplecov file to my main branch (Ubuntu VM) Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage/coverage.xml. 1306 / 1833 LOC (71.25%) covered Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage. 1306 / 1833 LOC (71.25%) covered. Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage/coverage.xml. 1306 / 1833 LOC (71.25%) covered Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage. 1306 / 1833 LOC (71.25%) covered. Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage/coverage.xml. 1306 / 1833 LOC (71.25%) covered Coverage report generated for /bin/bash ./tests/tests.sh to /root/projects/nautical-backup/coverage. 1306 / 1833 LOC (71.25%) covered. ```

This is leading my to believe the issue is with WSL2. In retrospect, it might have been a little crazy of me to develop a Bash project on Windows.....

I can move my development and tests to full Ubuntu and that should mitigate the problem. I can leave the GitHub issue open if you want to try to figure out why WSL2 is having these issues.

tomeon commented 10 months ago

@Minituff

In retrospect, it might have been a little crazy of me to develop a Bash project on Windows.....

Not terribly crazier than developing a Bash project, full stop :stuck_out_tongue_winking_eye:

Would be interesting to know whether the list of files with coverage stats changes between runs. If you've got jq available, this command will show you the names of files with coverage data:

matt@srv1 ~/.../Minituff/nautical-backup :) $ jq --raw-output '.[].coverage | keys[]' < ./coverage/.resultset.json
/path/to/Minituff/nautical-backup/pkg/backup.sh
/path/to/Minituff/nautical-backup/pkg/entry.sh
/path/to/Minituff/nautical-backup/pkg/env.sh
/path/to/Minituff/nautical-backup/pkg/logger.sh
/path/to/Minituff/nautical-backup/pkg/test.sh
/path/to/Minituff/nautical-backup/pkg/utils.sh
/path/to/Minituff/nautical-backup/snippets/docker-run-semver-example.sh
/path/to/Minituff/nautical-backup/snippets/docker-run-semver-major-example.sh
/path/to/Minituff/nautical-backup/tests/tests.sh
/path/to/Minituff/nautical-backup/tests/validate_dockerfile.sh
Minituff commented 10 months ago

Ha very true, its too late to turn back now 😜.

Looks like the tested files are the same each time.

I did have to modify your .simplecov because I could not get Ruby to recognize my ENV variable.

# .simplecov
SimpleCov.profiles.define 'bashcov' do
  load_profile 'rails'
  command_name 'Unit Tests'
  enable_coverage :branch
  primary_coverage :branch
  add_filter %r{^/snippets/}
  add_filter %r{^/.git/}
  add_filter %r{^/tests/}
  add_filter "pkg/test.sh"
  add_group "Pkg scripts", "/pkg"

  # simplecov 0.22.0+
  enable_coverage_for_eval if respond_to? :enable_coverage_for_eval
end

# Conditional loading of profiles
if ENV.key? 'SKIP_PROFILE'
  puts "Skipping bashcov profile..."
else
  puts "Loading bashcov configurations..."
  SimpleCov.load_profile 'bashcov'
end

Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup/coverage/coverage.xml. 0 / 385 LOC (0.00%) covered; 0 / 0 BC (100.00%) covered
Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup/coverage. 0 / 385 LOC (0.0%) covered.

/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/backup.sh
/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/entry.sh
/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/env.sh
/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/logger.sh
/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/utils.sh

Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup/coverage/coverage.xml. 256 / 402 LOC (63.68%) covered; 0 / 0 BC (100.00%) covered
Coverage report generated for /bin/bash tests/tests.sh to /mnt/c/Users/James-PC/IdeaProjects/nautical-backup/coverage. 256 / 402 LOC (63.68%) covered.

/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/backup.sh
/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/entry.sh
/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/env.sh
/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/logger.sh
/mnt/c/Users/James-PC/IdeaProjects/nautical-backup/pkg/utils.sh
tomeon commented 10 months ago

@Minituff -- the BASHCOV_COMMAND_NAME environment variable is set by Bashcov itself, so should activate the bashcov profile when Bashcov is running (and only when Bashcov is running, unless something else sets it for some reason?).

I don't have a WSL2 environment to investigate this on, but perhaps you could try running Bashcov without a .simplecov in place? This could help isolate the source of the issue. You could also check whether you see coverage data fluctuations when using other shell coverage tools (e.g. kcov). Finally, you could try saving the coverage data from one run (e.g., mv coverage{,.last}), re-run Bashcov, and then diff the results (e.g. diff coverage{,.last}/.resultset.json).

dalisoft commented 4 months ago

Hey @infertux I got there debugging values and repo at there

BASHCOV>fa26f0a5-fe3f-465c-96ee-a0600be01651573fa26f0a5-fe3f-465c-96ee-a0600be01651./bash_unitfa26exit 0
+ bashcov: warning: encountered an error parsing Bash's output (error was: expected integer for LINENO, got "uname -m\n_source/bash/release-me/release.sh"). This can occur if your script or its path contains the sequence "fa26f0a5-fe3f-465c-96ee-a0600be01651", or if your script unsets LINENO. Aborting early; coverage report will be incomplete.
Run completed using bashcov 3.1.2 with Bash 3.2, Ruby 3.3.1, and SimpleCov 0.22.0
Coverage report generated for /bin/bash ./bash_unit tests/cli/core.test.sh tests/cli/no-git.test.sh tests/gpg-sign/git-gpg-sign.test.sh tests/plugins/changelog.test.sh tests/plugins/docker.test.sh tests/plugins/git.test.sh tests/plugins/github-release.test.sh tests/plugins/npm-post.test.sh tests/plugins/npm.test.sh tests/plugins/template.test.sh tests/presets/conventional-commits.test.sh tests/presets/workspace.cargo-undefined.test.sh tests/presets/workspace.cargo.test.sh tests/presets/workspace.nodejs-undefined.test.sh tests/presets/workspace.nodejs.test.sh tests/presets/workspace.python-undefined.test.sh tests/presets/workspace.python.test.sh tests/presets/workspace.undefined.test.sh to /Users/dalisoft/Desktop/open_source/bash/release-me/coverage. 307 / 548 LOC (56.02%) covered.

Repo lives at here and i'm running command via env (cat .env.test) bundle exec bashcov ./bash_unit tests/**.test.sh locally