No coverage being created suddenly

[bjornbouetsmith]

Hi,

I am using altcover to cover my project and suddenly without me changing anything in any project files - coverage has stopped being generated:

The last working build: https://github.com/bjornbouetsmith/ROOT.Zfs/actions/runs/3503109293

Outputs

1>BuildProject:
     Build completed.
AltCoverRunPreparation:
         Creating folder /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/__Instrumented_ROOT.Zfs.Tests/
         Instrumenting files from /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/
         Writing files to /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/__Instrumented_ROOT.Zfs.Tests/
            => /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/ROOT.Zfs.Tests.dll
            => /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/ROOT.Zfs.Core.dll
            => /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/ROOT.Zfs.Public.dll
            => /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/AltCover.Monitor.dll

         Coverage Report: /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/coverage.opencover.xml

             /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/__Instrumented_ROOT.Zfs.Tests/ROOT.Zfs.Public.dll
                         <=  ROOT.Zfs.Public, Version=1.1.4.0, Culture=neutral, PublicKeyToken=null
             /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/__Instrumented_ROOT.Zfs.Tests/AltCover.Monitor.dll
                         <=  AltCover.Monitor, Version=8.3.0.0, Culture=neutral, PublicKeyToken=null
             /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/__Instrumented_ROOT.Zfs.Tests/ROOT.Zfs.Core.dll
                         <=  ROOT.Zfs.Core, Version=1.1.4.0, Culture=neutral, PublicKeyToken=null
             /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/__Instrumented_ROOT.Zfs.Tests/ROOT.Zfs.Tests.dll
                         <=  ROOT.Zfs.Tests, Version=1.1.4.0, Culture=neutral, PublicKeyToken=null
       AltCoverVSTestCore:
         Settings Before: 
         Settings After: /tmp/tmpiniL8o.altcover.runsettings
Test run for /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/__Instrumented_ROOT.Zfs.Tests/ROOT.Zfs.Tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:   667, Skipped:     0, Total:   667, Duration: 587 ms - /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/__Instrumented_ROOT.Zfs.Tests/ROOT.Zfs.Tests.dll (net6.0)
       AltCoverGenerateCoverageResult:
         ... /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/coverage.opencover.xml.0.acv (12,909b)
         75,642 visits recorded in 00:00:00.0100979 (7,490,864 visits/sec)
         A total of 75,642 visits recorded
         Coverage statistics flushing took 0.17 seconds
         Visited Classes 204 of 221 (92.31)
         Visited Methods 598 of 688 (86.92)
         Visited Points 4172 of 4987 (83.66)
         Visited Branches 945 of [156](https://github.com/bjornbouetsmith/ROOT.Zfs/actions/runs/3503109293/jobs/5867926284#step:10:157)1 (60.54)
         Maximum CRAP score [172](https://github.com/bjornbouetsmith/ROOT.Zfs/actions/runs/3503109293/jobs/5867926284#step:10:173).15

         ==== Alternative Results (includes all methods including those without corresponding source) ====
         Alternative Visited Classes 234 of 251 (93.23)
         Alternative Visited Methods 698 of 808 (86.39)
         Alternative maximum CRAP score 172.15
     1>Done Building Project "/home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/ROOT.Zfs.Tests.csproj" (VSTest target(s)).

The failing build and all subsequent ones goes straight from "Build completed"

     1>BuildProject:
         Build completed.

       AltCoverVSTestCore:
         Settings Before: 
         Settings After: /tmp/tmp1d5k0Z.altcover.runsettings
Test run for /home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/bin/Debug/net6.0/ROOT.Zfs.Tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:   677, Skipped:     0, Total:   677, Duration: 867 ms - ROOT.Zfs.Tests.dll (net6.0)
     1>Done Building Project "/home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/ROOT.Zfs.Tests.csproj" (VSTest target(s)).

So something makes altcover not prepare and instrument the assemblies being tested.

Can you come up with a reason why it would suddenly start without me changing any code?

I have checked the .NET version but both the old working and new failing are using the same version: 6.0.304, same with sonarscanner which is also being used in the build - and that version has also not changed. Also if I copy the test step to another action - it also fails.

When I run the exact same command line on my developer machine - it just works and altcover as expected outputs

 AltCoverRunPreparation

So if I had to guess - the issue is probably that the "AltCoverRunPreparation" target is somehow not being run, but I have no idea how it suddenly happened.

[bjornbouetsmith]

I can see that github switched from ubuntu 20.04 -> 22.04 - forcing ubuntu 20.04 does not change anything unfortunately

[bjornbouetsmith]

Their ubuntu image versions changed from: https://github.com/actions/runner-images/blob/ubuntu20/20221027.1/images/linux/Ubuntu2004-Readme.md To https://github.com/actions/runner-images/blob/ubuntu20/20221119.2/images/linux/Ubuntu2004-Readme.md

And the only thing that springs to my mind is that .NET 6 was upgraded from 6.0.402 -> 6.0.403 and that net 7.0.100 was added - so perhaps the .NET 7 runtime is somehow making things not work?

[bjornbouetsmith]

I will try to update to VS 17.4 where .NET 7 is included - to see if I can reproduce it locally - and I will also update altcover - I can see you have released a new version that mentions .net 7.

[bjornbouetsmith]

Updating to VS 17.4 which includes .NET 7 makes coverage not being collected as well. Will now update altcover to latest version to see if that fixes the issue

[bjornbouetsmith]

okay - updating altcover to latest version: 8.5.841 does not change anything - so I am blocked now - it seems that .NET 7 breaks gathering of code coverage from altcover.

[SteveGilham]

This looks like the known issue with dotnet test using the 7.0.100 runtime -- see microsoft/vstest#4014 -- which loses the /p:whatever arguments to dotnet test.

Since the latest runtime on the machine is used by default, until 7.0.101, which is promised to contain a fix for this regression, arrives, options are (in order of how cumbersome they are)

1. opt out of using the latest runtime with e.g. a global.json like

   {
   "sdk": {
   "version": "6.0.100",
   "rollForward": "latestMinor"
   }
   }

   to stick with the known-good v6

2. inject the properties as environment variables instead. As you have the 8.5.841 release, you can take advantage of the new features as documented in the extended release notes if you are using the build scripting support; manually assembled command lines need to be worked around manually, though.
3. use the AltCover command line tool and dotnet test --no-build

[bjornbouetsmith]

Its unfortunate that github just installed .NET SDK 7 on all their runner images without any way of specifying that you want to use an older image.

But also unfortunate that microsoft did not fix the issue even when it was reported before the official release.

I just tested with the global.json - and it works kind of - on my developer machine I get code coverage generated - on github - with the exact same command line arguments - nada - still no code coverage.

Very strange.

But at least I have some way of moving forward - now I just need to find the "right" solution until either MS fixes the SDK - or github rolls back to not including .NET SDK 7 in all images.

Thank you very much

[SteveGilham]

In your github actions, you can install the SDK of your choice via the setup-dotnet action like

    - uses: actions/setup-dotnet@v3
      with:
        dotnet-version: '6.0.403'

and lock to that.

As used the the AltCover build here

[bjornbouetsmith]

I have this:

      with:
        dotnet-version: 6.0.x

Which should be similar - but I think my issue is that I had placed the global.json in a too deep directory - when in the "root" of the git repository it seems to be picked up correctly and code coverage are now being generated again :-)

So thank you very much