search

coverlet-coverage / coverlet

Cross platform code coverage for .NET
MIT License
2.93k stars 385 forks source link

Coverlet throws BadImageFormatException #1612

Open AdityaKSemwal opened 4 months ago

AdityaKSemwal commented 4 months ago

Hi,

I am trying to run Coverlet but it throws the following error for every dll BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if the module has got local source. image

I have sets of e-2-e tests that can be run only through executable application. here is the command I am following coverlet D:\d\Source\myProject\bin\Debug --target "C:\Program Files\myProject.exe" --output "D:\coverlet.xml" --format opencover

I can open exe and run tests but I am not getting any code coverage. Any Idea how to fix this issue? I m using coverlet.console on .net framework project

Bertk commented 4 months ago

Please read the documentation and use the proposed parameters.

coverlet /path/to/test-assembly.dll --target "dotnet" --targetargs "test /path/to/test-project --no-build"

Usage: coverlet [arguments] [options]

Arguments:
  <ASSEMBLY|DIRECTORY>                  Path to the test assembly or application directory.

Options:
  -t|--target (REQUIRED)                Path to the test runner application.
  -a|--targetargs                       Arguments to be passed to the test runner.
  -o|--output                           Output of the generated coverage report
…
daveMueller commented 4 months ago

I think you are doing everything correct as described here for e-2-e tests. But I must admit I haven't used coverlet for integration tests in years. Without having something to debug it is really hard to say why this exception is being thrown. Which version of coverlet are you using and are you working with full or portable pdbs? Maybe this issue https://github.com/coverlet-coverage/coverlet/issues/1348 is related and you can try out the solution there. Nevertheless a repro would really help to analyze this.

AdityaKSemwal commented 4 months ago

I did more investigation and understood that BadImageFormatException is not an issue. It is also thrown for normal unit tests but I get coverage reports. The only problem I am facing is Coverlet is not gathering live coverage after a tool startup the executable.

most probably it is caused by an error [0212/191135.687:ERROR:gpu_process_transport_factory.cc(1029)] Lost UI shared context. DevTools listening on ws://127.0.0.1:8088/devtools/browser/210e2f33-0b05-4860-8927-54e36a43910e

also it takes by default candidate libraries which is not mentioned anywhere

AdityaKSemwal commented 4 months ago

@daveMueller thanks for your response. But the link you gave is not my issue. I managed to make it work for normal unit tests as i was missing nunit test adapter. but still struggling with e-2-e tests. Everything is fine but it doesn't have code coverage

daveMueller commented 4 months ago

The link I posted has nothing to do with normal unit tests (Code Coverage for integration tests and end-to-end tests). I can remember that I have done this at least once for e-2-e tests with Selenium. If you can provide a simple repro with the tools you are using, I could give it a try myself.

AdityaKSemwal commented 4 months ago

I don't have repro, I am using Autodesk Revit.exe as a target. and I have an add-in that runs tests through this application.

so here is the scenario

  1. open revit.exe
  2. run the addin
  3. Select the test dll
  4. tests are running through testrunner(class from nunit.core)

I can run the revit.exe with coverlet console command but it fails to collect the coverage. Same thing I do with opencover and it works. but unfortunately, I can't use opencover due to some restrictions.

daveMueller commented 4 months ago

OK I see, nothing I can reproduce by myself. Can you post a detailed output here? Maybe we can find some hints there. Just set the verbosity level to detailed. (troubleshoot)

AdityaKSemwal commented 4 months ago

This the log of e-2-e tests

C:\Users\Asemwal>coverlet D:\d\RevitCloudPlugIn\bin\Debug --target "C:\Program Files\Autodesk\Revit 2021\Revit.exe" --output "D:\coverlet.xml" --format opencover --include "[RVCCLBL*]*" --verbosity detailed
Included module filter '[RVCCLBL*]*'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\AL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\BLDimaTools.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\Castle.Core.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\CLBL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\CLIL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\CLUnitTest.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\DMBL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\DMIL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\DMUnitTests.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\DotNetZip.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ECBL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ECIntegrationTests.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ECTL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ELBL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ELIL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ELIntegrationTests.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ELUnitTests.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\EntityFramework.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\EntityFramework.SqlServer.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLCalcHolders.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLCommon.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLHeatingCooling.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLPlatform.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLRedimensioning.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLTapWater.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLIntegrationTestRevit.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLUnitTest.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCloud.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCLULCalculation.dll'
[0216/123237.157:ERROR:gpu_process_transport_factory.cc(1029)] Lost UI shared context.
DevTools listening on ws://127.0.0.1:8088/devtools/browser/f6aefc4a-7038-4e4b-b810-df0251bf48a1
Number of candidate libraries: 2
Candidate library: C:/Program Files (x86)/Common Files/Autodesk Shared/Materials/2020/assetlibrary_base.adsklib; version: 10
Candidate library: C:/Program Files (x86)/Common Files/Autodesk Shared/Materials/2021/assetlibrary_base.adsklib; version: 11
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Users\Asemwal
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

Calculating coverage result...
Hits file:'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLCalcHolders_4075763a-8d9b-4326-90b1-2639875a8458' not found for module: 'RVCCLBLCalcHolders'
Hits file:'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLCommon_4075763a-8d9b-4326-90b1-2639875a8458' not found for module: 'RVCCLBLCommon'
Hits file:'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLHeatingCooling_4075763a-8d9b-4326-90b1-2639875a8458' not found for module: 'RVCCLBLHeatingCooling'
Hits file:'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLPlatform_4075763a-8d9b-4326-90b1-2639875a8458' not found for module: 'RVCCLBLPlatform'
Hits file:'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLRedimensioning_4075763a-8d9b-4326-90b1-2639875a8458' not found for module: 'RVCCLBLRedimensioning'
Hits file:'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLTapWater_4075763a-8d9b-4326-90b1-2639875a8458' not found for module: 'RVCCLBLTapWater'
  Generating report 'D:\coverlet.xml'
+-----------------------+------+--------+--------+
| Module                | Line | Branch | Method |
+-----------------------+------+--------+--------+
| RVCCLBLCalcHolders    | 0%   | 0%     | 0%     |
+-----------------------+------+--------+--------+
| RVCCLBLCommon         | 0%   | 0%     | 0%     |
+-----------------------+------+--------+--------+
| RVCCLBLHeatingCooling | 0%   | 0%     | 0%     |
+-----------------------+------+--------+--------+
| RVCCLBLPlatform       | 0%   | 0%     | 0%     |
+-----------------------+------+--------+--------+
| RVCCLBLRedimensioning | 0%   | 0%     | 0%     |
+-----------------------+------+--------+--------+
| RVCCLBLTapWater       | 0%   | 0%     | 0%     |
+-----------------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

This is normal unit tests

C:\Users\Asemwal>coverlet D:\d\RevitCloudPlugIn\bin\Debug\RVCCLUnitTest.dll --target "dotnet" --targetargs "test D:\d\RevitCloudPlugIn\bin\Debug\RVCCLUnitTest.dll --no-build /TestAdapterPath:"D:\d\MechQuality24Q1\_external\nuget\nes\NUnitTestAdapter.2.3.0\build"" --output "D:\coverlet.xml" --format opencover --include "[RVCCLBL*]*" --verbosity detailed
Included module filter '[RVCCLBL*]*'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\AL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\BLDimaTools.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\Castle.Core.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\CLBL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\CLIL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\CLUnitTest.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\DMBL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\DMIL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\DMUnitTests.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\DotNetZip.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ECBL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ECIntegrationTests.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ECTL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ELBL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ELIL.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ELIntegrationTests.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\ELUnitTests.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\EntityFramework.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\EntityFramework.SqlServer.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLCalcHolders.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLCommon.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLHeatingCooling.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLPlatform.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLRedimensioning.dll'
BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if module has got local source.
Instrumented module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLBLTapWater.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCCLIntegrationTestRevit.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCloud.dll'
Excluded module: 'D:\d\RevitCloudPlugIn\bin\Debug\RVCLULCalculation.dll'
The following arguments have been ignored : "--no-build"
Microsoft (R) Test Execution Command Line Tool Version 17.8.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Passed!  - Failed:     0, Passed:  1006, Skipped:     0, Total:  1006, Duration: 6 s - RVCCLUnitTest.dll (net48)

Calculating coverage result...
Hits file:'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLCalcHolders_d5c8c3c2-98a6-4518-b07e-f192a7608eed' not found for module: 'RVCCLBLCalcHolders'
Hit file 'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLCommon_d5c8c3c2-98a6-4518-b07e-f192a7608eed' deleted
Hit file 'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLHeatingCooling_d5c8c3c2-98a6-4518-b07e-f192a7608eed' deleted
Hit file 'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLPlatform_d5c8c3c2-98a6-4518-b07e-f192a7608eed' deleted
Hit file 'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLRedimensioning_d5c8c3c2-98a6-4518-b07e-f192a7608eed' deleted
Hit file 'C:\Users\Asemwal\AppData\Local\Temp\RVCCLBLTapWater_d5c8c3c2-98a6-4518-b07e-f192a7608eed' deleted
  Generating report 'D:\coverlet.xml'
+-----------------------+--------+--------+--------+
| Module                | Line   | Branch | Method |
+-----------------------+--------+--------+--------+
| RVCCLBLCalcHolders    | 0%     | 0%     | 0%     |
+-----------------------+--------+--------+--------+
| RVCCLBLCommon         | 56.07% | 52.14% | 59.39% |
+-----------------------+--------+--------+--------+
| RVCCLBLHeatingCooling | 92.79% | 94.47% | 94.2%  |
+-----------------------+--------+--------+--------+
| RVCCLBLPlatform       | 47.67% | 0%     | 50%    |
+-----------------------+--------+--------+--------+
| RVCCLBLRedimensioning | 3.89%  | 1.47%  | 17.74% |
+-----------------------+--------+--------+--------+
| RVCCLBLTapWater       | 66.14% | 77.08% | 82.17% |
+-----------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 66.73% | 57.8%  | 70.52% |
+---------+--------+--------+--------+
| Average | 44.42% | 37.52% | 50.58% |
+---------+--------+--------+--------+

what ever action is done on revit.exe it doesn't collect code coverage

daveMueller commented 4 months ago

Sorry can't really identify what the problem is. Like you already mentioned, this looks a bit suspicious.

[0216/123237.157:ERROR:gpu_process_transport_factory.cc(1029)] Lost UI shared context.
DevTools listening on ws://127.0.0.1:8088/devtools/browser/f6aefc4a-7038-4e4b-b810-df0251bf48a1

As also mentioned in the documentation, coverlet is currently a bit limited for e-2-e tests and relies on a graceful process shutdown. There is some more information about this in those two issues #1079, #781. You probably can try out the coverage tool of MS that has a bit more advanced possibilities for e-2-e. https://github.com/coverlet-coverage/coverlet/issues/781#issuecomment-1340707747

Any other ideas @Bertk @MarcoRossignoli?

Bertk commented 4 months ago

@daveMueller I never tried to generate source code coverage metrics for e2e tests. We used a test concept which defined a test pyramid and source code coverage was mandatory for unit test and unit integration tests.

test pyramid was similar to this.

image
lineburg commented 2 months ago

Castle.Core.dll might be causing your problem. I got this error when a project that was not... suddenly WAS referencing it. It is not necessarily the fact that you reference Castle.Core.dll but might be the wrong "bitness". "BadImageFormatException is usually due to bitness. One project is 64 bit and the the other is 32 bit. Make sure both project target the same bitness x86 or 64."