search

Azure / azure-sdk-tools

Tools repository leveraged by the Azure SDK team.
MIT License
111 stars 176 forks source link

Testing framework for powershell scripts #4882

Open sima-zhu opened 1 year ago

sima-zhu commented 1 year ago

Background

No test coverage for our powershell scripts is long existing issue. We currently work on adding pester tests and testing in local. Several drawbacks for the local tests:

  1. It is not a convention for team member to add test coverage for everything they added.
  2. Forgot running the tests in local for new changes might introduce errors in tests, then the efforts of writing tests will be in vain.

    Goals

    We can have DevOps pipeline triggered every time we change the scripts. The tests coverage plan:

  3. We can cover modularized function tests first.
  4. Better to have integration tests on single script file.
  5. Have integration tests across multiple files.

    Proposals

    Running tests in pipeline would be the best choices.

    Proposal 1(Optimal)

    Test structure

    image

    Triggers Every time we touch scripts under eng/, the devOps testing pipeline would run.

    Proposal 2

    Test structure

    image

Triggers Every time we touch single scripts under eng/, the individual testing pipeline would run. If pipeline doesn't exist, then we use pipeline-generation create one.

sima-zhu commented 1 year ago

@weshaggard @benbp @danieljurek The proposals are currently under review and open to any suggestions. Could you please help brainstorm on a better plan?

benbp commented 1 year ago

We already have pester tests in the non-common eng directories of the language repositories as well, right? It would be nice if the pipeline testing steps are modular enough that we could add them into language pipelines easily (I'm not sure the right place yet).

benbp commented 1 year ago

My initial inclination is to start with just one pipeline that discovers all pester tests in the repository, as opposed to more directory specific tests, both for simplicity and because we have a lot of cross-directory script dependencies that aren't easily tracked.

weshaggard commented 1 year ago

My initial inclination is to start with just one pipeline that discovers all pester tests in the repository, as opposed to more directory specific tests

I agree with one pipeline but I would still stick to limiting to a given area not looking for all pester tests across the entire repo. For that reason I sort of lean towards proposal 1 with a scripts-tests directory that we scope the pester tests that we want to run to.

sima-zhu commented 1 year ago

We have common tests pipeline live now.

commit: https://github.com/Azure/azure-sdk-tools/commit/617ea8253052911f26cda0db15603622d0459ef6

pipeline: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2088942&view=logs&j=c7677f77-9c7a-5d06-5685-83acb5683895&t=358adba7-a1e6-5484-d654-f81d8812d6aa

sima-zhu commented 1 year ago

Next step is to have eng/scripts-tests pipelines for tools, net repo