Closed gregoks closed 4 years ago
So if I understand correctly Stryker fails to read the assembly information from VersionInfo.cs
? Would the project you ran this on happen to be open source? Or could you provide a reproduction project for us? That would help a lot to tackle this issue :+1:
@gregoks For the sake of testing, could you set the [assembly: AssemblyVersion("26.3.2")] to 26.4.1 as well and see if this fixes the mono.cecil problem?
It's not open source, one of the company that I work for internal assembly so unfortunately I cannot share it :( I have tried changing the version to 26.4.1 and it still gave me the same error, now it just cant find this version as well:
Failed to resolve assembly: 'MyService.ServicesInfra.StateMachines, Version=26.4.1.0, Culture=neutral, PublicKeyToken=null'
It does continues (as it did before) and only actually fails on the first time it starts writing:
[14:11:59 WRN] Stryker.NET encountered an compile error in C:\GIT\DefaultCollection\MyService\service\Service\ContractModelMapping\ContractModelMapper.cs (at 39:22) with message: The type or namespace name 'MerchantVerificationRequest' could not be found (are you missing a using directive or an assembly reference?) (Source code: MerchantVerificationRequest)
Perhaps that's what actually bothering him? Again its weird that it cant find this namespace because it exists and should compile as usual. I have ran stryker on several other solutions and it worked fine, only happens in this one.
Aha, I see. Then the issue is not related to Mono.Cecil and the Mono.Cecil error is just confusing noise.
Instead there seems to be a problem with either type resolution or we are introducing a breaking mutation that we are unable to remove. Either way I don't think we can investigate this problem without a reproduction unfortunately. These problems are usually specific to some configuration or syntax kind that we do not properly support.
I have a similar but more general error in another .net framework 4.6.1 solution that I have tried to run, it fails to find the Enum and again it compiles from visual studio:
[16:24:20 WRN] Stryker.NET encountered an compile error in C:\GIT\DefaultCollection\MyService\api\Service\Exceptions\BadRequestException.cs (at 13:15) with message: The type 'Enum' is defined in an assembly that is not referenced. You must add a reference to assembly 'mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e, Retargetable=Yes'. (Source code: LogLevel)
[16:24:20 WRN] Safe Mode! Stryker will try to continue by rolling back all mutations in method. This should not happen, please report this as an issue on github with the previous error message.
[16:24:20 FTL] Stryker.NET could not compile the project after mutation. This is probably an error for Stryker.NET and not your project. Please report this issue on github with the previous error message.
[16:24:20 ERR] An error occurred during the mutation test run
System.ApplicationException: Internal error due to compile error.
at Stryker.Core.Compiling.RollbackProcess.RemoveMutantIfStatements(SyntaxTree originalTree, ICollection`1 diagnosticInfo, Boolean devMode) in d:\a\1\s\src\Stryker.Core\Stryker.Core\Compiling\RollbackProcess.cs:line 193
at Stryker.Core.Compiling.RollbackProcess.Start(CSharpCompilation compiler, ImmutableArray`1 diagnostics, Boolean devMode) in d:\a\1\s\src\Stryker.Core\Stryker.Core\Compiling\RollbackProcess.cs:line 60
at Stryker.Core.Compiling.CompilingProcess.TryCompilation(MemoryStream ms, CSharpCompilation compilation, EmitResult previousEmitResult, Boolean devMode, Int32 retryCount) in d:\a\1\s\src\Stryker.Core\Stryker.Core\Compiling\CompilingProcess.cs:line 120
at Stryker.Core.Compiling.CompilingProcess.Compile(IEnumerable`1 syntaxTrees, MemoryStream ms, Boolean devMode) in d:\a\1\s\src\Stryker.Core\Stryker.Core\Compiling\CompilingProcess.cs:line 88
at Stryker.Core.MutationTest.MutationTestProcess.Mutate() in d:\a\1\s\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestProcess.cs:line 124
at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options, IEnumerable`1 initialLogMessages) in d:\a\1\s\src\Stryker.Core\Stryker.Core\StrykerRunner.cs:line 85
[16:24:20 INF] Time Elapsed 00:01:27.3706275
Which command do you run to try to compile the solution?
To answer your question: Compilation is performed internally using Roslyn so it can be done in memory and avoid modifying actual source files. Could you please try with the latest version (V0.14.2) that has been released today? There was a bug in V0.14.1 regarding .Net Framework selection that could explain what is happening with your projects.
Unfortunately I have tried using 0.14.2 version. Please see the attached log file, let me know if it helps you to figure out the issue. log-20191128.txt
Thanks, I think we have a clear signal of what went wrong:
2019-11-28T17:29:29.4790980+02:00 [VRB] Runner 3: [Information] Test run will use DLL(s) built for framework .NETFramework,Version=v0.0 and platform X86. Following DLL(s) do not match framework/platform settings.
DLL(s) built for framework .NETFramework,Version=v0.0 We need to understand why we identified 0.0 as the target version.
Great! :) Let me know if I can give you any more info, I really count on stryker.net for the infra in my organization and really like this to be fixed.
My bad, this line comes from VsTest, not Stryker core logic. Still analyzing your log
@gregoks the problem is not with compiling necessarily but rather with compiling after placing the mutations.
In your first solution it seems to me like we are placing a bad mutation that we cannot roll back for some reason, or we're missing a project reference.
In your second solution we are failing to get a reference to mscorlib (the core library for dotnet) which means all system types like Enum are missing when we try to compile the mutated source code.
In both cases we already know that your unmodified solution compiles successfully because Stryker always starts by compiling your unmodified sources.
Unfortunately this is a rather magic part of Stryker as we use a library called Buildalyzer for the dependency resolving. Msbuild and dependency resolving is very complicated in dotnet full framework.. This makes the problem hard to analyze even if we had a reproduction and almost impossible without a repro.
Thanks guys!
If you are saying that perhaps specific mutation caused it, I have tried running it by excluding all mutations but same error still happens, I ran setting the following flag:
--excluded-mutations "['string', 'logical','arithmetic','equality','boolean','unary','assignment','bitwise','update','checked','linq']"
We designed that feature to exclude mutations you don't want, not mutations that don't work. So that actually does place the mutations in the source code but then marks them as excluded. This so that we can display to the user which mutations they decided to skip.
Actually, this issue is not linked to any specific mutation. It turns out that Stryker fails to build the project using its internal logic, w/o mutations. This issue is probably linked to dependance resolution failing somewhere, but logging is lacking in that area. I am looking into the code to see if this situation can be improved.
The error message (Stryker.NET could not compile the project after mutation.) is not valid in this situation as there is no mutant left in the file (see the log).
Yeah perhaps if you can add more logs so I will run it again and see if we will have more information..?
I have cloned your code and ran it on my machine, I have changed this code in class CompilingProcess.
from this:
var compilation = CSharpCompilation.Create(analyzerResult.Properties.GetValueOrDefault("TargetName"),
syntaxTrees: trees,
options: new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary,
allowUnsafe: true,
cryptoKeyFile: analyzerResult.SignAssembly ? analyzerResult.AssemblyOriginatorKeyFile : null,
strongNameProvider: analyzerResult.SignAssembly ? new DesktopStrongNameProvider() : null,
nullableContextOptions: NullableContextOptions.Enable),
references: _input.AssemblyReferences);
to this:
var op = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary);
op = op.WithAssemblyIdentityComparer(DesktopAssemblyIdentityComparer.Default);
var compilation = CSharpCompilation.Create(analyzerResult.Properties.GetValueOrDefault("TargetName"),
syntaxTrees: trees,
options: op,
references: _input.AssemblyReferences);
based on several suggestions like this one:
now it actually compiled and worked. I'm no expert at all on the CSharpCompilation code and just made a shot in the dark, but does this change makes sense to you?
Nice find, i'll see what the impact of that change is on our integration test projects. I can't seem to find any information at all about that that actually does though... The docs contain no information at all.
Thanks! I have managed to successfully build the 2nd project (With the enum errors) but still no luck with the first project (the MyService.ServicesInfra.StateMachines, Version=26.3.2.0 error), I'll play with it some more, I'm guessing its also some sort of compilation configuration that might be altered?
@Mobrockers: I looked into the source code of DesktopAssemblyIdentityComparer, and it implements a replacement logic when expected version is not found (for well known assemblies). Default AssemblyComparer probably does not even try. This will fix this issue and a bunch of others!
@gregoks : regarding your original issue, StrykerMutator should dump the source code that failed compilation, could you please provide it, or the log as you did for the second one?
great! For the 1st error, I have made the following change and it helped, in EmbeddedResourcesGenerator class, I changed this code:
return ModuleDefinition.ReadModule(
assemblyPath,
new ReaderParameters(ReadingMode.Immediate)
{
InMemory = true,
ReadWrite = false
});
to this code:
return ModuleDefinition.ReadModule(
assemblyPath);
not sure if you need the readerParameter and not use the default one?
, I dont see the 26.3.2 error anymore though I saw in stryker the following log which I'm not sure is so good:
4 mutants got status CompileError. Reason: Could not compile
I will try to upload the log file of my initial failed run.
Thanks for this info as well! I will not comment as I am not familiar with the ressource generator design.
Regarding the message:
4 mutants got status CompileError. Reason: Could not compile
This is perfectly normal and can be ignored. As mutations happens at the source level; mutations can lead to non compilable code (the most frequent one is illegal arithmetic on string variables). Therefore Stryker contains rollback logic to remove the offending mutants.
Compile errors for mutants are fine. We just remove them and carry on. Some mutations we create just are not valid syntax, but it is harder to detect invalid syntax than it is to just remove the faulty mutation.
@Mobrockers : a suggestion: it could be nice to have a wiki page regarding the compilation errors, or a FAQ. What do you think?
great! For the 1st error, I have made the following change and it helped, in EmbeddedResourcesGenerator class, I changed this code:
return ModuleDefinition.ReadModule( assemblyPath, new ReaderParameters(ReadingMode.Immediate) { InMemory = true, ReadWrite = false });
to this code:
return ModuleDefinition.ReadModule( assemblyPath);
not sure if you need the readerParameter and not use the default one?
, I dont see the 26.3.2 error anymore though I saw in stryker the following log which I'm not sure is so good:
4 mutants got status CompileError. Reason: Could not compile
I will try to upload the log file of my initial failed run.
It's been a long time since I worked with this but from what I remember I don't think we can load the embedded resources if we deferr the module loading because by the time we compile the project the resource streams are closed (not 100% sure would have to get back into it. Mono.Cecil documentation is annoyingly sparse)
great! For the 1st error, I have made the following change and it helped, in EmbeddedResourcesGenerator class, I changed this code:
return ModuleDefinition.ReadModule( assemblyPath, new ReaderParameters(ReadingMode.Immediate) { InMemory = true, ReadWrite = false });
to this code:
return ModuleDefinition.ReadModule( assemblyPath);
not sure if you need the readerParameter and not use the default one? , I dont see the 26.3.2 error anymore though I saw in stryker the following log which I'm not sure is so good: 4 mutants got status CompileError. Reason: Could not compile I will try to upload the log file of my initial failed run.
It's been a long time since I worked with this but from what I remember I don't think we can load the embedded resources if we deferr the module loading because by the time we compile the project the resource streams are closed (not 100% sure would have to get back into it. Mono.Cecil documentation is annoyingly sparse)
Please check because it seemed fine when I ran it after the change
So when should we see new version with these fixes?:)
@gregoks Do you use embedded resources like language translation files? If not, then it is no surprise it works fine for you :p
@dupdob
@Mobrockers : a suggestion: it could be nice to have a wiki page regarding the compilation errors, or a FAQ. What do you think?
Yes, great idea
@gregoks Do you use embedded resources like language translation files? If not, then it is no surprise it works fine for you :p
@dupdob
@Mobrockers : a suggestion: it could be nice to have a wiki page regarding the compilation errors, or a FAQ. What do you think?
Yes, great idea
Question is why does it think that the referenced dll is an embedded resource? Perhaps you can add flag for those that dont have any embedded resources :)
If a user has language files as embedded resources, then every single test hitting a piece of code referencing the language strings will fail, causing major false positives. As a protection we try to load every single embedded resource available in the original DLL, this is the easiest way to load them. We do not know if there are embedded resources before we load them from the dll.
A flag to disable this behaviour is an idea, but really this feature should be transparant to the user and should just work.
So when should we see new version with these fixes?:)
See https://github.com/stryker-mutator/stryker-net/pull/828
You can download the preview package from here: https://dev.azure.com/stryker-mutator/Stryker/_packaging?_a=package&feed=stryker-net%40Local&package=dotnet-stryker&protocolType=NuGet&version=0.14.2-alpha.27&view=overview
You can install it on your system using:
install dotnet-stryker -g --version 0.14.2-alpha.27 --add-source https://pkgs.dev.azure.com/stryker-mutator/Stryker/_packaging/stryker-net%40Local/nuget/v3/index.json
@dupdob Could you test this also on some of your projects?
So the 0.14.2-alpha.27 fixed the "Enum" error (2nd solution) and its working fine so yay! :)
I have ran it again using the 0.14.2-alpha.27 version on the first solution with the failed to resolve assembly error. Still getting the error. please see the attached log file. log-20191129.txt
@gregoks Yes, a fix for that problem is not in 0.14.2-alpha.27
Yeah just wanted to run it on the latest version. I think I found the problem. for some reason I had in my ProjectUnderTest used internal project that was both referenced as project reference and regular reference and I guess it caused stryker problem because it didnt put it under referenced projects so he didnt find any types from it.
I'll do some more testing but it looks ok so far :D
@gregoks Are you saying the embedded resources problem is no longer relevant? If so can I close this issue since we fixed the other problem?
Yep looks like it. Ill let you know if this happens again.
Thanks for the quick responses guys!
Closed by #828
Hi,
When running stryker using the following command:
"C:\Program Files\dotnet\dotnet.exe" stryker --solution-path C:\MyPath\MyName.sln --project-file Service.csproj --test-project-file UnitTests.csproj --reporters "['html', 'progress']" --excluded-mutations "['string', 'logical']" --coverage-analysis perTest --threshold-high 90 --threshold-low 75 --threshold-break 0 --log-level debug
The compilation fails with the following errors:The assembly that it fails to resolve for some reason has the following definitions in the versionInfo.cs :
Expected behavior The compilation should be successful as it succeeds from visual studio and It should search for version 26.4.1 as the assemblyFileVersion and not 26.3.2.
Desktop (please complete the following information):