Pester (In-)Compatibility Not Correctly Handled

bormm commented 4 years ago

Where are you running it?

Azure DevOps Service (VSTS) + Self Hosted Agent with Windows 10 + Visual Studio

Version of Extension/Task

Version 10.0.5

Expected behaviour and actual behaviour

I found out only by reading the source code, the Task >= V10 does only support Pester V5. The input fields do not mention any Pester V5 requirement, but also mentioned Pester V4 is part of the task, which is of course not the case and also incompatible. The build in is 5.0.1.

If specifying preferredPesterVersion V3.40.0, the task fails with an error due the unknown type:

[error]Der Typ [PesterConfiguration] wurde nicht gefunden.

[error]In D:\azdevops-build_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1:143 Zeichen:43

$result = Invoke-Pester -Configuration ([PesterConfiguration]$Pester ...

Steps to reproduce the problem

Specify preferredPesterVersion V3.40.0.

Suggestion

There is a version check in Pester.ps1, which only opt-outs V4:

if ($TargetPesterVersion -match '^4') {
    Write-Host "##vso[task.logissue type=error]This version of the task does not support Pester V4, please use task version 9."
    exit 1
}

A idea would be the following:

if ([Version]$TargetPesterVersion -lt [Version]"5.0.0") {
    Write-Host "##vso[task.logissue type=error]This version of the task does not support Pester below V5. Please use task version 9."
    exit 1
}

That would also not use string operations for versions.

The description of "TargetPesterVersion" task.json should be updated, maybe its also possible to do some verification there.

ChrisLGardner commented 4 years ago

You're right that the version handling isn't great, I want to support people wanting to run the task with semver specified and the version type doesn't support that. I'll check for not matching 5 or starting with 3 or 4 instead.

The help etc should definitely be updated to make it more clear that it's 5+ only.

bormm commented 4 years ago

You mean something like "1.0.0-alpha+001" ?

ChrisLGardner commented 4 years ago

Yes, Pester has been doing releases of beta versions for a while now so ideally we'd support people running them through this task like that. I think there's a bit more work I need to do to support that anyway so I'll see what I can do this week to fix this issue.

bormm commented 4 years ago

@ChrisLGardner I understand. Then maybe this is a highlight for you. Unfortunately only > PS 6.2

ChrisLGardner commented 4 years ago

Yeah, that's why I'm avoiding using it since a lot of people will still be on 5.1 (or earlier). I could make use of it and try/catch it but I'd rather not. String parsing isn't the ideal approach but with the versions of PS I have to support and wanting to keep it simple this is about the best I can do.

bormm commented 4 years ago

@ChrisLGardner Anyway: Unfortunately I also can't use Pester V5 due the missing parameter feature they dropped. So I also have to use V9 of your Task.

ChrisLGardner commented 4 years ago

Yeah, that's why both are available since the way to interact with Invoke-pester has changed and that isn't an option right now, so I've got both 9 and 10 available with different flows to handle those differences.

I'll get this versioning thing fixed this week for both versions of the task.

bormm commented 4 years ago

Ok, great. So V9 is not completely dropped? Would be great to have both "engines" in the same task and its chosen by the selected Pester Version.

ChrisLGardner commented 4 years ago

V9 is the way to use pester 4.x (and 3 in theory but I would migrate off that). As there won't be much dev happening on pester 4 I'd rather not have to deal with both approaches to calling pester in the same task. It adds a lot of extra logic that isn't needed and makes ongoing maintenance harder.

pester / AzureDevOpsExtension