search

b3b00 / coverlet-action

coverlet github action
MIT License
5 stars 4 forks source link

Coverage Output File Needs to exist? #6

Closed agray closed 3 years ago

agray commented 3 years ago

I have this in pipeline.yml file:

- name: Run Unit Tests
  uses: b3b00/coverlet-action@1.1.1
    with:
      testProject: ${{ env.UT_PROJECTDIR }}
      output: 'opencover.info'
      outputFormat: 'opencover'
      excludes: '[program]*,[UnitTests]Tests.*,[Merlin]*'

env.UT_PROJECTDIR is defined as:

UT_PROJECTDIR: UnitTests\\ # unit test directory

The output I get is:

Run b3b00/coverlet-action@1.1.1 create coverlet args found exclusions [program],[UnitTests]Tests.,[Merlin] coverlet args -p:coverletOutput=opencover.info -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:Exclude=\"[program],[UnitTests]Tests.,[Merlin]\" UnitTests\ run dotnet test Determining projects to restore... All projects are up-to-date for restore. Saucery -> /home/runner/work/Saucery/Saucery/Saucery/bin/Debug/net5.0/Saucery.dll UnitTests -> /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll Test run for /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll (.NETCoreApp,Version=v5.0) Microsoft (R) Test Execution Command Line Tool Version 16.11.0 Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait... A total of 1 test files matched the specified pattern. Skipped FlowControlTest [< 1 ms]

Passed! - Failed: 0, Passed: 41, Skipped: 1, Total: 42, Duration: 829 ms - /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll (net5.0)

Error: error occured : coverage file not found at ./opencover.info [1]covergae file not found at ./opencover.info setting output coverageFile : >>./opencover.info<< coverlet test done.

I interpret the error message as requiring the output file to exist !? I would have expected this output file to be WRITTEN by the action and it SHOULDN'T exist.

This doesn't seem right. What am I missing? Must be something silly on my part.

Thanks in advance.

b3b00 commented 3 years ago

hello @agray , output file is not required to exist. I think the problem is witn the testprohect arg of the action. it needs to be the path to the unit test csproj file. in your case it only contains the directory name. it should be somethin like 

-testproject: UnitTests/UnitTests.csproj

hopes this will help.

agray commented 3 years ago

Unfortunately it still fails:

- name: Run Unit Tests
  uses: b3b00/coverlet-action@1.1.1
    with:
      testProject: ${{ env.UT_PROJECT }}
      output: 'opencover.info'
      outputFormat: 'opencover'
      excludes: '[program]*,[UnitTests]Tests.*,[Merlin]*'

env.UT_PROJECT is defined as:

UT_PROJECT: UnitTests/UnitTests.csproj # unit test directory

And my output is:

Run b3b00/coverlet-action@1.1.1 create coverlet args found exclusions [program],[UnitTests]Tests.,[Merlin] coverlet args -p:coverletOutput=opencover.info -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:Exclude=\"[program],[UnitTests]Tests.,[Merlin]\" UnitTests/UnitTests.csproj run dotnet test Determining projects to restore... All projects are up-to-date for restore. Saucery -> /home/runner/work/Saucery/Saucery/Saucery/bin/Debug/net5.0/Saucery.dll UnitTests -> /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll Test run for /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll (.NETCoreApp,Version=v5.0) Microsoft (R) Test Execution Command Line Tool Version 16.11.0 Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait... A total of 1 test files matched the specified pattern. Skipped FlowControlTest [< 1 ms]

Passed! - Failed: 0, Passed: 41, Skipped: 1, Total: 42, Duration: 818 ms - /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll (net5.0)

Error: error occured : coverage file not found at UnitTests/opencover.info [1]covergae file not found at UnitTests/opencover.info setting output coverageFile : >>UnitTests/opencover.info<< coverlet test done.

agray commented 3 years ago

Can we please have this reopened.

b3b00 commented 3 years ago

@agray i' ve reopened the issue. could you point me to you repo so i can look closer at your workflow

agray commented 3 years ago

Thank you @b3b00

https://github.com/Sauceforge/Saucery/actions

b3b00 commented 3 years ago

@agray I've published a new release of coverlet-action (1.1.2.2) with debug information. Could you try it (I've tried on a fork but I miss some secrets)

b3b00 commented 3 years ago

Finally igot my own saucage api user and keys. And I've found your issue. You miss a coverlet package reference in the unittests.csproj

You can check it at this commit : https://github.com/b3b00/Saucery/commit/72838716ed658a735f1fa2b3aeff0982218ee536

And matching pipeline run : https://github.com/b3b00/Saucery/actions/runs/1277666968

agray commented 3 years ago

Thank you @b3b00 Here is my run (still in progress at time of writing): https://github.com/Sauceforge/Saucery/runs/3718732765?check_suite_focus=true

Not sure why it is taking 8 minutes and counting on the unit test step though. Hmm.

Looks like it waiting for a hosted runner:

2021-09-27T09:43:46.5932871Z Found online and idle hosted runner in the current repository's organization account that matches the required labels: 'ubuntu-latest' 2021-09-27T09:43:46.7026437Z Waiting for a Hosted runner in the 'organization' to pick this job... 2021-09-27T09:43:47.1011290Z Job is waiting for a hosted runner to come online. 2021-09-27T09:43:51.3109154Z Job is about to start running on the hosted runner: Hosted Agent (hosted)

agray commented 3 years ago

Cancelled and reran build. Worked that time. https://github.com/Sauceforge/Saucery/runs/3718847744?check_suite_focus=true Thank you @b3b00, I was clearly the PEBCAK ;)

Question: Does the action support failing the build if the coverage is too low? Can that be configured in the pipeline yml?

b3b00 commented 3 years ago

i ve just added a threshold param to set a minimal coverage. i ve not test it though you can have a try with last release. check readme for documentation.

agray commented 3 years ago

Have look at this: https://github.com/Sauceforge/Saucery/runs/3718980448?check_suite_focus=true

Specifically: image

treshold, should be threshold I think.

b3b00 commented 3 years ago

@agray thank you for testing. it should be fixed with release 1.1.3

agray commented 3 years ago

Is this what I'm supposed to see?

https://github.com/Sauceforge/Saucery/runs/3719548987?check_suite_focus=true

b3b00 commented 3 years ago

Error message is not really explicit but lowering threshold to 10 make the build pass. I've used the default coverlet threshold mechnism and error message seems to "disappear". For now it's working but I will later look at a way to make a clear error message or return a output variable stating that there is a coverage threshold error.

agray commented 3 years ago

Thank you @b3b00 for your assistance.

The exception threw me (no pun intended). Was expecting a "clean" error message to indicate build failure.

https://github.com/Sauceforge/Saucery/runs/3719660453?check_suite_focus=true (I need more unit tests :/)

b3b00 commented 3 years ago

I guess that the issue comes from github actions runner that does not display standard error stream. coverlet display an error message like this : 

error : The minimum line coverage is below the specified 80

on standard error output so we don't see it :( I am trying to find a way to redirect stderr to stdout. If I can not do it I will look at a way to manage myself the threshold not relying on coverlet own mechanism

b3b00 commented 3 years ago

@agray I 've just published coverlet-action 1.1.6 that solve the threshold issue. the error message is now easier to understand.

agray commented 3 years ago

@b3b00 Thank you.