search

stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.76k stars 175 forks source link

Official Stryker CI tool installer tasks #195

Closed rouke-broersma closed 4 years ago

rouke-broersma commented 5 years ago

Most CI's available have some form of tool installer plug-in support. Since we have tool installation support that is independent of project package references we could support such build tasks to install stryker in the CI pipeline. This would mean that stryker can be run on projects that might not necessarily have a dependency on stryker. This would also increase the visibility of stryker. All the build task would have to do is dotnet tool install StrykerMutator.DotNetCoreCli --tool-path <build agent workdir tool path> --version <version chosen to install in build>

It is important that --tool-path is used and not -g as we absolutely do not want a global install on a build agent.

For vsts see: https://github.com/Microsoft/vsts-task-tool-lib https://github.com/Microsoft/vsts-task-tool-lib/blob/master/docs/overview.md https://github.com/Microsoft/vsts-task-tool-lib/blob/master/sample.ts

Any other platforms are welcome as well I'm sure. If for example anyone is interested in creating a jenkins tool installer plugin I will be able to give some guidance.

nicojs commented 5 years ago

This is a great idea! I like the idea of the tool just being a wrapper around the CLI. That way we are sure to keep them in sync.

How would you provide arguments to the stryker task? Could we start by just passing them along? This is the same approach we took for stryker-cli and grunt-stryker. It saves you the trouble of constantly updating the package whenever new command line arguments are available.

rouke-broersma commented 5 years ago

In the first place this plugin only has to provide the install functionality in the same way for example the vstest testplatform installer task works. The only configuration value for the task would be the version to install (minimum version being the version where we introduce dotnet tool install) in the form of a dropdown with available versions including 'Latest'. The task needs to set the installed plugin version number so we can use it later. We could later introduce another task (in the same package) that can run stryker. This task would have fields for every configuration option available to the installed version.

nicojs commented 5 years ago

This task would have fields for every configuration option available to the installed version.

Why not 1 field with "arguments" and a link to an examples page? That would be easier to maintain.

rouke-broersma commented 5 years ago

Because then the task would provide no added value over just using the dotnet or even commandline tasks with arguments. I think it is more user friendly to provide configuration fields so the user knows what configuration is available to the chosen version. It lowers the bar to entry.

ptoonen commented 5 years ago

Starting work on this.

rouke-broersma commented 5 years ago

@ptoonen Be aware that we do not currently support dotnet global tools, see #192 We had implemented global tool support but then found out that support for project-based cli tools is not compatible with global tool support. We have not yet decided whether we want to switch to global tool, support both variants or stick with project-level cli tool only.

Personally I have the feeling microsoft is going to 'drop' support for project-based cli tools and will move to global tools only in the future. Perhaps you have some insights?

@richardwerkman What do you think?

Once we make a decision it should be trivial to release a global tool package, probably as a separate csproj.

richardwerkman commented 5 years ago

For me supporting both project-based and global would be best. Whatever microsoft will choose we will support later on.

ptoonen commented 5 years ago

@Mobrockers I don't think they'll be dropping per-project extension any time soon. However, I do think the recommended way is to go global. And I'm not alone in that.

The problem is that global tools also introduce issues with multiple developers as @nicojs raises in #192 Imagine dev number 1 installed version X and dev number 2 installed version Y. On the build server it's not really much of an issue due to it being installed for the user running the build and on top of that, we can specify in the task which version (range) to use.

Now the great part is that there is a thing coming called 'Repo Tools' - see the post I referenced above. This should solve these issues. In the meanwhile we there are three approaches that I see for the tasks:

  1. We employ the same method as SonarQube does, employing msbuild targets to inject the references to test-projects. We'd need to properly identify the projects or ask for user input to specify the project(s) they want to install the DotNetCliToolReference to.
  2. We only allow running on 'Stryker Enabled' projects for now.
  3. We introduce a stand-alone global tool for now

In any case, tasks can be made and changed later on to a newer version so that's not the issue at hand :-) Option 2 seems to be the easiest option, option 1 is also relatively easy to do. I would avoid number 3 because we'd either be deprecating this method soon(ish) or we'd have to support it for a long time. @richardwerkman this also means that supporting both ways puts us in a bit of a pickle.

ptoonen commented 5 years ago

O, on a related note, let's monitor and explore this issue.

rouke-broersma commented 4 years ago

Stryker has been refactored to a dotnet core tool. A task specifically for installing stryker in the build is not neccesary at this time. I would expect most users will migrate to local tools as well starting with dotnet core 3.0 and we will start suggesting the local tool as the recommended way of installing stryker.