stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.78k stars 185 forks source link

Unexpected behaviour depending on calling shell #1826

Closed dancingCamel closed 2 years ago

dancingCamel commented 2 years ago

Describe the bug Running dotnet stryker with a config file including "since":{"target":"master", "enabled":"true"} returns different numbers of files changed depending on the calling shell. Git BASH - every file in project appears as changed Windows Powershell - only changed files appear as changed cmd - only changed files appear as changed husky pre-push script calling powershell script which runs dotnet styker - every file in project appears as changed Powershell in Azure pipeline - every file in project apprears as changed.

Logs Powershell:

2021-11-30T11:15:40.7858634+00:00  [DBG] Matched with upstream canonical name "refs/heads/master" (2b1a95cc)
2021-11-30T11:15:44.1762359+00:00  [INF] 18 files changed (8364427c)
...
2021-11-30T11:15:44.1981098+00:00  [INF] Total number of tests found: "455". (a6b4f85b)
2021-11-30T11:15:44.1996267+00:00  [INF] Initial testrun started. (4c47b8e8)
2021-11-30T11:15:44.2011695+00:00  [VRB] VsTest run settings set to: "<RunSettings>

Git Bash:

2021-11-30T11:08:55.6791189+00:00  [DBG] Looking for branch matching "master" (d7e261c8)
2021-11-30T11:08:55.7099321+00:00  [DBG] Matched with upstream canonical name "refs/heads/master" (2b1a95cc)
2021-11-30T11:08:59.5842670+00:00  [INF] 858 files changed (8364427c)
...
2021-11-30T11:08:59.6006454+00:00  [INF] Total number of tests found: "455". (a6b4f85b)
2021-11-30T11:08:59.6006939+00:00  [INF] Initial testrun started. (4c47b8e8)
2021-11-30T11:08:59.6017437+00:00  [VRB] VsTest run settings set to: "<RunSettings>

Expected behavior Since target should only pick up the files that have actually been changed regardless of the calling environment

Desktop (please complete the following information):

Additional context husky pre-push script:

#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

cwd=$(pwd)

powershell ./testrunner.ps1 \(\'project.name\'\)  $cwd

testrunner.ps1 (located in repo root):

param ([Parameter(Mandatory)]$projects,[Parameter(Mandatory)]$basePath)

# Examples for parameters
# -Projects ('project_name', 'project.name') 
# -BasePath "C:\some_path_here\repo_name" 

function RunForOneAssembly ($csprojPath, $testPath ) {
    Write-Host "csprojPath: " $csprojPath
    Write-Host "Moving to test directory: "$testPath
    Set-Location $testPath

    Write-Host "Calling Tests"
    dotnet test 

    if( -not $? ){
        Write-Host "Error in running tests, exiting the script"
        exit 1;
    }

    Write-Host "Calling Stryker"

    $strykerCommand="dotnet stryker"

    Write-Host $strykerCommand

    Invoke-Expression $strykerCommand

    if( -not $? ){
        Write-Host "Error in running Stryker, exiting the script"
        exit 1;
    }
}

function RunTests ($projects, $basePath) {
    $startDir = Get-Location

    $projects | ForEach-Object  {
        $testPath = "$basePath\tests\$_.Tests\"
        $project_file = "$basePath\src/$_\$_.csproj"
        $csprojPath = "$basePath\src\$_\$_.csproj"
        RunForOneAssembly $csprojPath $testPath
    }

    Set-Location $startDir
}

Write-Host "Running test script"

RunTests $projects $basePath 

stryker-config.json:

{
    "stryker-config": {
        "project": "proj.name\\proj.name.csproj",
        "target-framework": "net5.0",
        "mutation-level": "Standard",
        "reporters": ["dots", "html", "json"],
        "ignore-mutations": ["string"],
        "ignore-methods": null,
        "mutate": ["!**/Program.cs", "!**/Startup.cs"],
        "since": {
            "target": "master",
            "enabled": true
        },
        "thresholds": {
            "high": 90,
            "low": 80,
            "break": 70
        }
    }
}
rouke-broersma commented 2 years ago

Sounds like a problem with file paths and the unix emulation git bash performs.

dancingCamel commented 2 years ago

Sounds like a problem with file paths and the unix emulation git bash performs.

Yes, I think it's either the way git handles paths, or it's line endings. It's not exclusively git bash causing a problem - if a powershell script runs from inside a husky shell script, and when powershell runs on the pipeline we also get the same problem.

rouke-broersma commented 2 years ago

We use a library to interact with git. If this library and a specific shell don't work nicely together this should be fixed there.