Closed timandella closed 1 year ago
richardwerkman
commented
1 year ago In v3.5.0 we fixed another issue regarding embedded resources. So simply reverting the code is not desirable. We'll have to look into why this happens. @rouke-broersma do you maybe have an idea?
rouke-broersma
commented
1 year ago Yep, the new solution reads the csproj for embedded resources, this fails for wildcards because the current implementation assumes each embedded resources is a full reference to a single embedded resource. This is done here: https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/Initialisation/EmbeddedResourcesGenerator.cs#L102
We would have to perform a disk search using the contents of the Include/Update in the case of wildcards, to find all necessary embedded resources.
@timandella I assume your KQL queries are not actually used during unit testing, so in 3.4.0 it could very well be the case that embedded resources were already missing, but you never noticed it because we didn't crash. Is that correct or am I off track here?
timandella
commented
1 year ago The embedded resources are not actually missing. They are all present and correct in the folders. and we even have tests to check the text of the kql files to make sure they are not empty.
The issue is that in the .csrproj file, we include them by doing <EmbeddedResource Include="Queries\Shared\*.kql" /> as developers were always forgetting to set the Build Action in Visual Studio when a new one was added.
rouke-broersma
commented
1 year ago They are all present and correct in the folders. and we even have tests to check the text of the kql files to make sure they are not empty.
Sorry for the confusion, I meant missing during stryker compilation. The problem we tried to resolve is that embedded resources discovery during stryker compilation was failing often, so the embedded resources would be missing during a stryker run.
During your tests, do you actually use the embedded resource or do you open the file from disk?
timandella
commented
1 year ago We have a static class, with a property corresponding to each embedded resource
public static readonly string MyTestQuery = GetQuery("Queries.Shared.MyTestQuery.kql")
where GetQuery method reads the embeddedResource like so
private static string GetQuery(string name)
{
using var stream = _executingAssembly.GetManifestResourceStream(typeof(Queries), $"Queries.{name}.kql");
using var readers = new StreamReader(stream);
return readers.ReadToEnd();
}We then have tests that check the query is not empty (and also tests that actually parse them).
(I did just try deleting the contents of one for the .kql files, and the relevant test failed as expected).
rouke-broersma
commented
1 year ago Are any mutants generated for this code? The biggest problem is that a missing embedded resource causes a runtime exception during unit testing, which will cause the mutant to be flagged as killed. But that is a false positive, because it's only failing because the embedded resource is missing.
If you don't have any mutants in the code that uses embedded resources, you will never notice that the embedded resources were missing. It wouldn't matter, because there are no mutants to test anyway,
If you do have mutants in the code that uses embedded resources, could you try and remove the assertions in the test and see if the mutant still ends up killed? If it does, that confirms that embedded resources are actually missing.
timandella
commented
1 year ago This is what gets written to the console when dotnet-stryker 3.4.0 is installed:
2023-02-03T00:20:34.5415626Z ##[section]Starting: Run dotnet-stryker TestProject.Data.UnitTests
2023-02-03T00:20:34.5424871Z ==============================================================================
2023-02-03T00:20:34.5425177Z Task : PowerShell
2023-02-03T00:20:34.5425368Z Description : Run a PowerShell script on Linux, macOS, or Windows
2023-02-03T00:20:34.5425635Z Version : 2.212.0
2023-02-03T00:20:34.5425821Z Author : Microsoft Corporation
2023-02-03T00:20:34.5426025Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2023-02-03T00:20:34.5426323Z ==============================================================================
2023-02-03T00:20:34.7460065Z Generating script.
2023-02-03T00:20:34.7491386Z ========================== Starting Command Output ===========================
2023-02-03T00:20:34.7538095Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/a1e0377d-0ad9-42a3-a17f-0ec168073418.ps1'
2023-02-03T00:20:35.2358621Z
2023-02-03T00:20:35.2360687Z _____ _ _ _ _ ______ _______
2023-02-03T00:20:35.2361635Z / ____| | | | | \ | | ____|__ __|
2023-02-03T00:20:35.2362515Z | (___ | |_ _ __ _ _| | _____ _ __ | \| | |__ | |
2023-02-03T00:20:35.2363378Z \___ \| __| '__| | | | |/ / _ \ '__| | . ` | __| | |
2023-02-03T00:20:35.2364230Z ____) | |_| | | |_| | < __/ | | |\ | |____ | |
2023-02-03T00:20:35.2365093Z |_____/ \__|_| \__, |_|\_\___|_| (_)|_| \_|______| |_|
2023-02-03T00:20:35.2365950Z __/ |
2023-02-03T00:20:35.2366743Z |___/
2023-02-03T00:20:35.2367163Z
2023-02-03T00:20:35.2367526Z
2023-02-03T00:20:35.3073867Z Version: 3.4.0
2023-02-03T00:20:35.3074534Z
2023-02-03T00:20:35.6556813Z [00:20:35 INF] Identifying project to mutate.
2023-02-03T00:20:37.6722860Z [00:20:37 INF] The project /home/vsts/work/1/s/src/TestProject.Data/TestProject.Data.csproj will be mutated.
2023-02-03T00:20:39.6371756Z [00:20:39 INF] Analysis complete.
2023-02-03T00:20:39.6383647Z [00:20:39 INF] Building test project /home/vsts/work/1/s/src/TestProject.Data.Tests/TestProject.Data.UnitTests.csproj (1/1)
2023-02-03T00:20:46.4053206Z [00:20:46 INF] We could not locate a baseline for branch refs/heads/nut/package-update-C03C51C7, now trying fallback version refs/heads/main
2023-02-03T00:20:46.4361159Z [00:20:46 INF] We could not locate a baseline report for the current branch, version or fallback version. Now running a complete test to establish a fresh baseline.
2023-02-03T00:20:46.5272339Z [00:20:46 INF] 8 files changed
2023-02-03T00:20:46.5273465Z [00:20:46 INF] Changed file /home/vsts/work/1/s/nuget.config
2023-02-03T00:20:46.5274388Z [00:20:46 INF] Changed file /home/vsts/work/1/s/src/TestProject.Api/TestProject.Api.csproj
2023-02-03T00:20:46.5275465Z [00:20:46 INF] Changed file /home/vsts/work/1/s/src/TestProject.Data/TestProject.Data.csproj
2023-02-03T00:20:46.5276509Z [00:20:46 INF] Changed file /home/vsts/work/1/s/src/TestProject.Events/TestProject.Events.csproj
2023-02-03T00:20:46.5277414Z [00:20:46 INF] Changed file /home/vsts/work/1/s/src/nuget.config
2023-02-03T00:20:46.5278350Z [00:20:46 INF] Changed file /home/vsts/work/1/s/test/Test.TestProject.Common/Test.TestProject.Common.csproj
2023-02-03T00:20:46.5279669Z [00:20:46 INF] Changed file /home/vsts/work/1/s/test/Test.TestProject.Events/Test.TestProject.Events.csproj
2023-02-03T00:20:46.5280589Z [00:20:46 INF] Changed file /home/vsts/work/1/s/test/nuget.config
2023-02-03T00:20:46.5286792Z [00:20:46 INF] Total number of tests found: 242.
2023-02-03T00:20:46.5287658Z [00:20:46 INF] Initial testrun started.
2023-02-03T00:20:52.0764003Z [00:20:52 WRN] Stryker.NET encountered an compile error in /home/vsts/work/1/s/src/TestProject.Data/Readers/CustomerExperienceDataReader.cs (at 64:19) with message: Use of unassigned local variable 'netPromoterScoreModel' (Source code: netPromoterScoreModel)
2023-02-03T00:20:52.5846103Z [00:20:52 INF] 406 mutants created
2023-02-03T00:20:52.6169001Z [00:20:52 INF] 26 mutants got status CompileError. Reason: Mutant caused compile errors
2023-02-03T00:20:52.6170220Z [00:20:52 INF] 373 mutants got status Ignored. Reason: Mutant not changed compared to target commit
2023-02-03T00:20:52.6172213Z [00:20:52 INF] 7 mutants got status Ignored. Reason: Removed by method filter
2023-02-03T00:20:52.6173119Z [00:20:52 INF] 406 total mutants are skipped for the above mentioned reasons
2023-02-03T00:20:52.6173976Z [00:20:52 INF] 0 total mutants will be tested
2023-02-03T00:20:52.6177228Z [00:20:52 WRN] It looks like all mutants with tests were ignored. Try a re-run with less ignoring!
2023-02-03T00:20:52.6179020Z [00:20:52 WRN] It looks like all mutants resulted in compile errors. Mutants sure are strange!
2023-02-03T00:20:53.6206884Z
2023-02-03T00:20:53.6207712Z Killed: 0
2023-02-03T00:20:53.6208102Z Survived: 0
2023-02-03T00:20:53.6208469Z Timeout: 0
2023-02-03T00:20:53.6467956Z Hint: by passing "--open-report or -o" the report will open automatically once
2023-02-03T00:20:53.6468514Z Stryker is done.
2023-02-03T00:20:53.6468686Z
2023-02-03T00:20:53.6469082Z Your html report has been generated at:
2023-02-03T00:20:53.6504770Z file:///home/vsts/work/1/s/src/TestProject.Data.Tests/StrykerOu
2023-02-03T00:20:53.6505554Z tput/2023-02-03.00-20-35/reports/mutation-report.html
2023-02-03T00:20:53.6506069Z You can open it in your browser of choice.
2023-02-03T00:20:53.9567740Z [00:20:53 INF] Time Elapsed 00:00:18.4162052
2023-02-03T00:20:53.9572071Z [00:20:53 INF] Stryker was unable to calculate a mutation score
2023-02-03T00:20:54.7558077Z ##[section]Finishing: Run dotnet-stryker TestProject.Data.UnitTestsWhen dotnet-stryker 3.5.1 is used (with no changes to the code base being tested), the following to logged to the console
2023-02-08T00:21:53.1732040Z ##[section]Starting: Run dotnet-stryker TestProject.Data.UnitTests
2023-02-08T00:21:53.1741470Z ==============================================================================
2023-02-08T00:21:53.1741993Z Task : PowerShell
2023-02-08T00:21:53.1742191Z Description : Run a PowerShell script on Linux, macOS, or Windows
2023-02-08T00:21:53.1742457Z Version : 2.212.0
2023-02-08T00:21:53.1742635Z Author : Microsoft Corporation
2023-02-08T00:21:53.1742861Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2023-02-08T00:21:53.1743162Z ==============================================================================
2023-02-08T00:21:53.3714159Z Generating script.
2023-02-08T00:21:53.3724162Z ========================== Starting Command Output ===========================
2023-02-08T00:21:53.3751351Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/3ff4276f-c794-48e1-9cdc-a55c42ac46ae.ps1'
2023-02-08T00:21:53.8469173Z
2023-02-08T00:21:53.8480822Z _____ _ _ _ _ ______ _______
2023-02-08T00:21:53.8485339Z / ____| | | | | \ | | ____|__ __|
2023-02-08T00:21:53.8486443Z | (___ | |_ _ __ _ _| | _____ _ __ | \| | |__ | |
2023-02-08T00:21:53.8487280Z \___ \| __| '__| | | | |/ / _ \ '__| | . ` | __| | |
2023-02-08T00:21:53.8489451Z ____) | |_| | | |_| | < __/ | | |\ | |____ | |
2023-02-08T00:21:53.8491690Z |_____/ \__|_| \__, |_|\_\___|_| (_)|_| \_|______| |_|
2023-02-08T00:21:53.8494023Z __/ |
2023-02-08T00:21:53.8494991Z |___/
2023-02-08T00:21:53.8495363Z
2023-02-08T00:21:53.8500789Z
2023-02-08T00:21:53.9212838Z Version: 3.5.1
2023-02-08T00:21:53.9213618Z
2023-02-08T00:21:54.2237911Z [00:21:54 INF] Identifying project to mutate.
2023-02-08T00:21:56.2158101Z [00:21:56 INF] The project /home/vsts/work/1/s/src/TestProject.Data/TestProject.Data.csproj will be mutated.
2023-02-08T00:21:58.1309023Z [00:21:58 INF] Analysis complete.
2023-02-08T00:21:58.1318655Z [00:21:58 INF] Building test project /home/vsts/work/1/s/src/TestProject.Data.Tests/TestProject.Data.UnitTests.csproj (1/1)
2023-02-08T00:22:04.2394414Z [00:22:04 INF] We could not locate a baseline for branch refs/heads/nut/package-update-8B5F4CE4, now trying fallback version refs/heads/main
2023-02-08T00:22:04.2733416Z [00:22:04 INF] We could not locate a baseline report for the current branch, version or fallback version. Now running a complete test to establish a fresh baseline.
2023-02-08T00:22:04.3652540Z [00:22:04 INF] 6 files changed
2023-02-08T00:22:04.3654327Z [00:22:04 INF] Changed file /home/vsts/work/1/s/nuget.config
2023-02-08T00:22:04.3655824Z [00:22:04 INF] Changed file /home/vsts/work/1/s/src/TestProject.Api/TestProject.Api.csproj
2023-02-08T00:22:04.3658769Z [00:22:04 INF] Changed file /home/vsts/work/1/s/src/TestProject.Data/TestProject.Data.csproj
2023-02-08T00:22:04.3660347Z [00:22:04 INF] Changed file /home/vsts/work/1/s/src/nuget.config
2023-02-08T00:22:04.3661720Z [00:22:04 INF] Changed file /home/vsts/work/1/s/test/Test.TestProject.Common/Test.TestProject.Common.csproj
2023-02-08T00:22:04.3662654Z [00:22:04 INF] Changed file /home/vsts/work/1/s/test/nuget.config
2023-02-08T00:22:04.3663478Z [00:22:04 INF] Total number of tests found: 242.
2023-02-08T00:22:04.3664182Z [00:22:04 INF] Initial testrun started.
2023-02-08T00:22:09.9558585Z [00:22:09 WRN] Stryker.NET encountered an compile error in /home/vsts/work/1/s/src/TestProject.Data/Readers/CustomerExperienceDataReader.cs (at 64:19) with message: Use of unassigned local variable 'netPromoterScoreModel' (Source code: netPromoterScoreModel)
2023-02-08T00:22:10.3139666Z [00:22:10 INF] Time Elapsed 00:00:16.1662399
2023-02-08T00:22:10.3188969Z Unhandled exception. System.ArgumentNullException: Value cannot be null. (Parameter 'key')
2023-02-08T00:22:10.3189940Z at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
2023-02-08T00:22:10.3190981Z at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
2023-02-08T00:22:10.3191953Z at Stryker.Core.Compiling.RollbackProcess.Start(CSharpCompilation compiler, ImmutableArray`1 diagnostics, Boolean lastAttempt, Boolean devMode)
2023-02-08T00:22:10.3193233Z at Stryker.Core.Compiling.CsharpCompilingProcess.TryCompilation(Stream ms, Stream symbolStream, CSharpCompilation compilation, EmitResult previousEmitResult, Boolean lastAttempt, Int32 retryCount)
2023-02-08T00:22:10.3194423Z at Stryker.Core.Compiling.CsharpCompilingProcess.Compile(IEnumerable`1 syntaxTrees, Stream ilStream, Stream symbolStream)
2023-02-08T00:22:10.3195339Z at Stryker.Core.MutationTest.CsharpMutationProcess.CompileMutations()
2023-02-08T00:22:10.3196298Z at Stryker.Core.MutationTest.CsharpMutationProcess.Mutate()
2023-02-08T00:22:10.3197125Z at Stryker.Core.MutationTest.MutationTestProcess.Mutate()
2023-02-08T00:22:10.3197931Z at Stryker.Core.Initialisation.ProjectMutator.MutateProject(StrykerOptions options, IReporter reporters, IEnumerable`1 solutionProjects)
2023-02-08T00:22:10.3198857Z at Stryker.Core.Initialisation.ProjectOrchestrator.MutateProjects(StrykerOptions options, IReporter reporters)+MoveNext()
2023-02-08T00:22:10.3199654Z at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
2023-02-08T00:22:10.3200333Z at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
2023-02-08T00:22:10.3201152Z at Stryker.Core.StrykerRunner.RunMutationTest(IStrykerInputs inputs, ILoggerFactory loggerFactory, IProjectOrchestrator projectOrchestrator)
2023-02-08T00:22:10.3202086Z at Stryker.CLI.StrykerCli.RunStryker(IStrykerInputs inputs) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 93
2023-02-08T00:22:10.3202974Z at Stryker.CLI.StrykerCli.<>c__DisplayClass10_0.<Run>b__0() in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 68
2023-02-08T00:22:10.3203870Z at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass143_0.<OnExecute>b__0(CancellationToken _)
2023-02-08T00:22:10.3204788Z at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
2023-02-08T00:22:10.3205621Z at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-02-08T00:22:10.3206412Z at Stryker.CLI.StrykerCli.Run(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 74
2023-02-08T00:22:10.3207209Z at Stryker.CLI.Program.Main(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/Program.cs:line 14
2023-02-08T00:22:11.3342567Z ##[error]PowerShell exited with code '1'.
2023-02-08T00:22:11.3374231Z ##[section]Finishing: Run dotnet-stryker TestProject.Data.UnitTestsWhat I can do (tomorrow), is rustle up a test project to demo this (as I can't upload the current project code due to it being a company project)
rouke-broersma
commented
1 year ago @timandella that's not neccesary, I know what's happening and I know how to fix it. What I am curious about is if the previous embedded resources solution worked for your project at all, in that case we might want to use both methods to discover embedded resources. But to know that I need to know if the embedded resources are used during mutation testing in the first place. To determine this I have these questions about version 3.4.0:
If you look at the full mutation report (not from the pipeline as you seem to be using since and baseline mode), are there any teste mutants in the specific code that uses embedded resources. You don't have to share the code or report with me, I only need to know that there are mutants, and if they are killed, survived or both.
If there are any, could you try and remove the assertions in the test and see if the mutant result changes? Eg if any go from killed to survived.
timandella
commented
1 year ago My apologies for misunderstanding. I am fairly new to the world of Stryker (a colleague of mine set it up...)
By having looked a the full mutation report produced with the --log-to-file option, it appears there are no test mutants in the specific code that uses that reads the embedded resource. The offending "GetQuery" method is not mentioned once.
rouke-broersma
commented
1 year ago In that case I will assume that the old method is not useful and we simply need to support wildcards. Thank you!
danrobertsamenity
commented
1 year ago Does the wildcard issue only apply to embedded resources? I ask because I am getting the same error I do not see any embedded resources in our project that are using wildcards. We do however have this in one of our csproj files:
<ItemGroup>
<Content Include="Apis\*\*.jmespath" CopyToPublishDirectory="PreserveNewest" />
</ItemGroup>For the project we are having issues with, it is only EmbeddedResources where we used wildcards.
rouke-broersma
commented
1 year ago Does the wildcard issue only apply to embedded resources? I ask because I am getting the same error I do not see any embedded resources in our project that are using wildcards. We do however have this in one of our csproj files:
<ItemGroup> <Content Include="Apis\*\*.jmespath" CopyToPublishDirectory="PreserveNewest" /> </ItemGroup>
Mhm I'm not sure how that could be but I'll investigate.
Describe the bug I have a C# project that contains text files that is included in the the project with a Build Action of "Embedded Resource". Within the .csproj file, there is a wildcard used to capture all such files, rather than individually listing them
When running the "dotnet-stryker" command against the project, an unhandled exception occurs
Should file names be explicitly listed in .csproj, then no exception occurs, and Stryker runs as expected
Expected behavior No unhandled exception to occur
Desktop (please complete the following information):
Additional context Issue started happening in v3.5.0. It worked without exception in v3.4.0