search

stryker-mutator / mutation-testing-elements

🗃 A schema for mutation testing results with the web components to visualise it 📊
Apache License 2.0
28 stars 19 forks source link

Schema validation fails for schema version 2 #2493

Closed vbreuss closed 1 year ago

vbreuss commented 1 year ago

Describe the bug I can't use the dashboard reporter to upload reports since the tool was updated to 3.8.1. The same report with tool version 3.7.1 is working.

Logs I get the following error in the Github Build Pipeline:

All mutants have been tested, and your mutation score has been calculated
…[07:56:25 ERR] Failed to upload report to the dashboard at https://dashboard.stryker-mutator.io/api/reports/github.com/Testably/Testably.Architecture.Rules/main?module=Testably.Architecture.Rules
System.Net.Http.HttpRequestException: Response status code does not indicate success: [40](https://github.com/Testably/Testably.Architecture.Rules/actions/runs/4900499091/jobs/8751189918#step:5:41)0 (Bad Request).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Stryker.Core.Clients.DashboardClient.PublishReport(JsonReport report, String version)
[07:56:25 ERR] Uploading to stryker dashboard failed...`

When I revert back to v3.7.1 (dotnet tool install dotnet-stryker --tool-path ../tools --version 3.7.1) the report gets successfully uploaded:

All mutants have been tested, and your mutation score has been calculated
…Hint: by passing "--open-report:dashboard or -o:dashboard" the report will open 
automatically once Stryker is done.

Your report has been uploaded at:
https://dashboard.stryker-mutator.io/reports/github.com/Testably/Testably.Archit
ecture.Rules/topic/vb/change-stryker-version?module=Testably.Architecture.Rules
You can open it in your browser of choice.

Expected behavior The report should be uploaded to https://dashboard.stryker-mutator.io/.

Desktop (please complete the following information):

Additional context In my other project Stryker also no longer uploads the dashboard report, but it shows a different error message:

[16:04:59 INF] Initial testrun started.
[16:05:15 INF] Time Elapsed 00:01:13.8443548
Unhandled exception. System.InvalidOperationException: Sequence contains no matching element
   at System.Linq.ThrowHelper.ThrowNoMatchException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
   at Stryker.Core.Initialisation.ProjectMutator.EnrichTestProjectsWithTestInfo(InitialTestRun initialTestRun, TestProjectsInfo testProjectsInfo)
   at Stryker.Core.Initialisation.ProjectMutator.MutateProject(StrykerOptions options, IReporter reporters, IEnumerable`1 solutionProjects)
   at Stryker.Core.Initialisation.ProjectOrchestrator.MutateProjects(StrykerOptions options, IReporter reporters)+MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Stryker.Core.StrykerRunner.RunMutationTest(IStrykerInputs inputs, ILoggerFactory loggerFactory, IProjectOrchestrator projectOrchestrator)
   at Stryker.CLI.StrykerCli.RunStryker(IStrykerInputs inputs) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 93
   at Stryker.CLI.StrykerCli.<>c__DisplayClass10_0.<Run>b__0() in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 68
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass143_0.<OnExecute>b__0(CancellationToken _)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Stryker.CLI.StrykerCli.Run(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 74
   at Stryker.CLI.Program.Main(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/Program.cs:line 14
/home/runner/work/_temp/e6b4cac3-c1[27](https://github.com/Testably/Testably.Abstractions/actions/runs/4902459842/jobs/8754314766#step:5:28)-4f6d-b5f5-cf92c89680a7.sh: line 2:  1858 Aborted                 (core dumped) ../../tools/dotnet-stryker -f ../.github/stryker/Stryker.Config.Testing.json -v "${GITHUB_REF#refs/heads/}" -r "Dashboard" -r "cleartext"
Error: Process completed with exit code 1[34](https://github.com/Testably/Testably.Abstractions/actions/runs/4902459842/jobs/8754314766#step:5:35).
rouke-broersma commented 1 year ago

Looks like two different problems. The first looks like the dashboard has not yet been updated to support the new version of the report, the second seems to be some problem detecting your unit tests which we did not expect.

vbreuss commented 1 year ago

@rouke-broersma : Both issues started occurring with v3.8.1 and work well with the previous version.

As a workaround I reverted both projects to v3.7.1.

Should I split this issue in two?

rouke-broersma commented 1 year ago

Yes both changes that caused these two issues were released in 3.8. If you could create a new issue for the second problem that would be great!

vbreuss commented 1 year ago

I created stryker-mutator/stryker-net#2501 for the second issue!

vbreuss commented 1 year ago

@rouke-broersma , it seems this issue is still not resolved as of today. I reverted my change and use now the latest version 3.8.2, but the build still fails:

_____ _              _               _   _ ______ _______  
  / ____| |            | |             | \ | |  ____|__   __| 
 | (___ | |_ _ __ _   _| | _____ _ __  |  \| | |__     | |    
  \___ \| __| '__| | | | |/ / _ \ '__| | . ` |  __|    | |    
  ____) | |_| |  | |_| |   <  __/ |    | |\  | |____   | |    
 |_____/ \__|_|   \__, |_|\_\___|_| (_)|_| \_|______|  |_|    
                   __/ |                                      
                  |___/                                       

Version: 3.8.2

[08:41:29 INF] Identifying project to mutate.
[08:41:39 INF] Found project /home/runner/work/Testably.Architecture.Rules/Testably.Architecture.Rules/Source/Testably.Architecture.Rules/Testably.Architecture.Rules.csproj to mutate.
[08:41:39 INF] Building test project /home/runner/work/Testably.Architecture.Rules/Testably.Architecture.Rules/Tests/Testably.Architecture.Rules.Tests/Testably.Architecture.Rules.Tests.csproj (1/1)
[08:41:52 INF] Total number of tests found: 138.
[08:41:52 INF] Initial testrun started.
[08:41:58 INF] 299 mutants created
[08:41:58 INF] Capture mutant coverage using 'CoverageBasedTest' mode.
[08:42:00 INF] [12](https://github.com/Testably/Testably.Architecture.Rules/actions/runs/4903156687/jobs/8760595632#step:5:13)    mutants got status CompileError. Reason: Mutant caused compile errors
[08:42:00 INF] 52    mutants got status NoCoverage.   Reason: Not covered by any test.
[08:42:00 INF] 60    mutants got status Ignored.      Reason: Removed by block already covered filter
[08:42:00 INF] 124   total mutants are skipped for the above mentioned reasons
[08:42:00 INF] [17](https://github.com/Testably/Testably.Architecture.Rules/actions/runs/4903156687/jobs/8760595632#step:5:18)5   total mutants will be tested

All mutants have been tested, and your mutation score has been calculated
…[08:43:49 ERR] Failed to upload report to the dashboard at https://dashboard.stryker-mutator.io/api/reports/github.com/Testably/Testably.Architecture.Rules/main?module=Testably.Architecture.Rules
System.Net.Http.HttpRequestException: Response status code does not indicate success: 400 (Bad Request).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Stryker.Core.Clients.DashboardClient.PublishReport(JsonReport report, String version)
[08:43:49 ERR] Uploading to stryker dashboard failed...

[08:43:49 INF] Time Elapsed 00:02:[19](https://github.com/Testably/Testably.Architecture.Rules/actions/runs/4903156687/jobs/8760595632#step:5:20).81[20](https://github.com/Testably/Testably.Architecture.Rules/actions/runs/4903156687/jobs/8760595632#step:5:21)[21](https://github.com/Testably/Testably.Architecture.Rules/actions/runs/4903156687/jobs/8760595632#step:5:22)7
[08:[43](https://github.com/Testably/Testably.Architecture.Rules/actions/runs/4903156687/jobs/8760595632#step:5:44):49 INF] The final mutation score is 64.76 %
rouke-broersma commented 1 year ago

Mhm I'll take another look. We should probably also log the failure message instead of only the error code..

rouke-broersma commented 1 year ago 
2023-05-07T09:02:54.999789068Z: [ERROR]        'D:\\a\\Testably.Abstractions\\Testably.Abstractions\\Tests\\Testably.Abstractions.Compression.Tests\\ZipFile\\Tests.cs': [Object]
2023-05-07T09:02:54.999792968Z: [ERROR]      }
2023-05-07T09:02:54.999796068Z: [ERROR]    },
2023-05-07T09:02:54.999799268Z: [ERROR]    query: { module: 'Testably.Abstractions.Compression' },
2023-05-07T09:02:54.999802568Z: [ERROR]    params: {},
2023-05-07T09:02:54.999805768Z: [ERROR]    reqId: 'daf91545d8474aab90524d31b542122d',
2023-05-07T09:02:54.999808968Z: [ERROR]    time: 2023-05-07T09:02:54.995Z,
2023-05-07T09:02:54.999812268Z: [ERROR]    duration: 450,
2023-05-07T09:02:54.999815368Z: [ERROR]    error: {
2023-05-07T09:02:54.999818568Z: [ERROR]      name: 'BAD_REQUEST',
2023-05-07T09:02:54.999821768Z: [ERROR]      message: 'Invalid report. data/schemaVersion must match pattern "^1(\\.\\d*)?$"',
2023-05-07T09:02:54.999825268Z: [ERROR]      status: 400,
2023-05-07T09:02:54.999828268Z: [ERROR]      errors: [],
2023-05-07T09:02:54.999831568Z: [ERROR]      stack: 'BAD_REQUEST: Invalid report. data/schemaVersion must match pattern "^1(\\.\\d*)?$"\n' +
2023-05-07T09:02:54.999835069Z: [ERROR]        '    at ReportsController.verifyRequiredPutReportProperties (file:///app/node_modules/@stryker-mutator/dashboard-backend/dist/src/api/ReportsController.js:98:23)\n' +
2023-05-07T09:02:54.999839069Z: [ERROR]        '    at ReportsController.update (file:///app/node_modules/@stryker-mutator/dashboard-backend/dist/src/api/ReportsController.js:35:14)\n' +
2023-05-07T09:02:54.999842569Z: [ERROR]        '    at runMicrotasks (<anonymous>)\n' +
2023-05-07T09:02:54.999845969Z: [ERROR]        '    at processTicksAndRejections (node:internal/process/task_queues:96:5)'
2023-05-07T09:02:54.999849269Z: [ERROR]    }
2023-05-07T09:02:54.999852369Z: [ERROR]  }
rouke-broersma commented 1 year ago

The html elements schema version enforces version 1, but we have had a breaking change and are on version 2.

https://github.com/stryker-mutator/mutation-testing-elements/blob/e605f8d0d3caf8691390497b92f1cbaf158faec3/packages/report-schema/src/mutation-testing-report-schema.json#L15