Updates for cross-platform matrix testing

[bravo-kernel]

This updates the pipeline with a matrix for cross-platform testing with separated jobs for Code Coverage and Script Analysis. Fully documented step-by-step in azure-pipelines.yml but noticeable changes:

• uses gitversion dotnet instead of the previous task cannot run on Linux/Windows
• used direct Pester because the Black Marble task cannot run on Linux/Windows
• mimics the previous pipeline so the outcome should be identical (artifact etc)

FYI this matrix is based on a similar CI script from another language. I have deliberately tried to avoid using Marketplace Extensions/posh tasks where possible so that this yaml stays easy to reason about and contributors can make contributions without dependencies on the task/extension owner.

TODO

• [ ] fix gitversion dotnet
• [ ] remove Install-RequiredModule
• [ ] add Script Analysis

[bravo-kernel]

There seems to be a discrepancy between my reference-repo and this PR so I will need to look into that, please be patient.

[Jaykul]

I just merged a refactor that moved most of the yaml stuff to an external repo so we can share that with multiple projects. Are you on the PowerShell slack or discord -- we could discuss on #poshcode channel ;-)

[bravo-kernel]

I saw the refactor but am not sold on the idea to be honest. I see little point of moving logic to different repos when the yaml is so straight-forward and easy to reason about if things ever go amiss.

May I suggest fixing this PR so you can see the approach working, we could discuss to move things to that repo later. The functional goal is my main priority here.. making sure that the ModuleBuilder templates will compile on all platforms.

[Jaykul]

Knock yourself out. 😉

I still think the approach is wrong: we should not build and test N separate copies of the software on N operating systems, and then release only one, which was tested on only one operating system.

Instead, build one copy, and test it everywhere, so that the package you're releasing is the same package that was tested everywhere (and as I said above, running a full build of this repo as an additional test, to ensure we're producing the same output).

[bravo-kernel]

Not too fast, I do take your input more than seriously and it does makes sense, I am most likely just missing some implementation details so best to await your refactor and takes things from there. Shall I close this PR for now?

[Jaykul]

Well, if you are willing/interested, I'd love to have you figure out the matrix testing part 😁

I have a multi-job build now, with

• Gitversion
• Build (depends on gitversion)
• ScriptAnalyzer (depends on build)
• Test (depends on build) ...

If you look at ../Azure-Pipelines/Pester-job.yml what I do there is pull the build artifact and test it. What I want to do there is pull the build artifact onto each image in your matrix and test it:

    Linux:
      imageName: 'ubuntu-16.04'
    Mac:
      imageName: 'macOS-10.14'
    WindowsPS5:
      imageName: 'vs2017-win2016'
    WindowsPS6:
      imageName: 'vs2017-win2016'

Ideally, I'd like to run it on both PS5 and PS6 on Windows, but even though it slows us down, it might be worth testing coverage on all of them, just because if we ever have OS-specific code, we'll need the combination...

The ScriptAnalysis is already in a separate parallel job, so that can stay as-is, I think?

And as I said above, once we get that going, we probably want to add a test that does a build and compares the output to the code under test 😎

[bravo-kernel]

OK, I will take a look but first I need to figure out why our Azure UIs are displaying different information and producing different results. I was under the assumption a yaml would always yield the exact same Azure results regardless of the repo it is used in but somehow it appears that is not the case (here), including the failing setup.

Mine:

Yours:

[bravo-kernel]

It think I have something up and running that matches what you are describing above. I did have to enable the multi-stage preview feature in my Devops profile first but after that I was able to combine multi-stage with the cross-platform matrix which... I think is the way to go.

This is what it looks like when running the pipeline (with 5 fictional stages):

This is what it looks like after it is finished:

Generated using this yaml:

# Name of our build, the GitVersion variable will be replaced during one of the steps
name: $(Build.DefinitionName)_$(GitVersion_InformationalVersion)

# Determines which branch(es) will cause a CI build to be started
trigger:
- master

# Stages precede strategy, in other words each stage can contain a strategy (or multiple or none)
# Full run-cycle described here: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/runs?view=azure-devops
stages:
- stage: Prepare
  jobs:
  - job:
    pool:
      vmImage: 'ubuntu-16.04'
    steps:
    - powershell: |
        "OS = $($env:AGENT_OS)" | Out-Host
      displayName: 'Dummy Prepare Step'

- stage: Build
  jobs:
  - job:
    pool:
      vmImage: 'windows-2019'
    steps:
    - powershell: |
        "OS = $($env:AGENT_OS)" | Out-Host
      displayName: 'Dummy Build Step'

- stage: Test
  jobs:
  - job:
    strategy:
      matrix:
        Linux:
          imageName: 'ubuntu-16.04'
        Mac:
          imageName: 'macOS-10.14'
        Windows:
          imageName: 'windows-2019'
        Code Coverage:
          imageName: 'windows-2019'
        Script Analysis:
          imageName: 'windows-2019'
    pool:
      vmImage: $(imageName)
    steps:
    - powershell: |
        "OS = $($env:AGENT_OS)" | Out-Host
      displayName: 'Dummy Test Step'

- stage: Document
  jobs:
  - job:
    pool:
      vmImage: 'windows-2019'
    steps:
    - powershell: |
        "OS = $($env:AGENT_OS)" | Out-Host
      displayName: 'Dummy Document Step'

- stage: Release
  jobs:
  - job:
    pool:
      vmImage: 'windows-2019'
    steps:
    - powershell: |
        "OS = $($env:AGENT_OS)" | Out-Host
      displayName: 'Dummy Release Step'

Additional info

• The steps are dummies but you get the idea and I think you can probably integrate your existing yml-templates quicker than I can do trying
• I purposely left out dependsOn etc. to prevent clutter

Also, I think I can now better explain what I was trying to point out earlier. As the ModuleBuilder skeleton templates will use the exact same azure-pipelines.yml as used by ModuleBuilder itself (as discussed and which is a very good thing because will we only need to maintain a single source of truth) we do however basically enforce the end-user to develop on Windows due to the imageName we use for the Build stage. For now this is not even an issue so let's not get distracted and get this up-and-running first.

However, I think the solution for what I was suggesting earlier would be to (eventually/in the future) make the Build stage configurable by e.g. passing a vmImage parameter. This way:

1. ModuleBuilder always use Build stage using the current windows image (as you prefer and makes sense, a good thing!)
2. the end-user gets the exact same setup when creating a new module skeleton using the defaults and thus the windows build image (also a good thing!)
3. however, if the user for some reason desires to build his/her module on another platform they simply change the image name for the Build stage in their local copy of azure-pipelines.yml and their module would be built on that platform. Any custom alterations or requirements they need for that build could be done in the Preparations stage (or whatever it would be called).
4. this way we basically realize a generic/perfected pipeline once, let all end-users benefit

I hope you see the added value in that. For now I think it is a couple of steps too far to even implement so let's get the default up-and-running first.

[Jaykul]

I'm not sure why this never quite succeeded, but I added the cross-platform matrix to my shared templates, so I can updating my other modules to use the new version of the template.

[bravo-kernel]

Nice, hope your builds profit from it.