search

Azure / azure-sdk-tools

Tools repository leveraged by the Azure SDK team.
MIT License
114 stars 180 forks source link

Add support to chunk testing large sets of test projects and run them in parallel #2951

Open weshaggard opened 2 years ago

weshaggard commented 2 years ago

Today for things like the net - core pipelines we are testing a large number of projects in the same job which is only growing over time. Not only does that cause the pipeline to run slower and slower (currently about 60mins) but it makes retrying failures painful. To help with this we need to figure out a way to dynamically generate a test matrix that will allow us to run a subset of the full project list on independent agents.

This will allow us to increase the overall time and allow retrying to be much more efficient.

This chunking will be interesting for all the languages but the .NET core pipeline and soon the core mgmt pipeline will be needing this support to be able to maintain a reasonable testing time for PR validation.

weshaggard commented 2 years ago

@benbp I could use some ideas from you on how to achieve this goal.

benbp commented 2 years ago

@weshaggard FYI we just added functionality to support passing in pre steps which can be used to dynamically generate a test matrix before the job matrix is generated. @scbedd is using this here as an example.

weshaggard commented 2 years ago

PreStep might be interesting in selection but I think we will need an independent job in order to figure out how to generate a test matrix that is chunked across multiple jobs.

benbp commented 2 years ago

Is there any difference if it runs in a job vs. a step in the generate matrix job? The output is dynamic either way. Or do you mean we would run the generate matrix job multiple times, each with a different dynamic input matrix? (If so, then that could also be achieved with multiple matrix configs as input, but right now it's static via parameters).

weshaggard commented 2 years ago

I think this step could be done in the generate matrix job I think I miss-understood that change to have PreSteps into each of the jobs in the matrix. So that set of PreSteps could be the right extension point to implement this. We just need to figure out how to partition the work and so that each job knows the set of work it needs to do.

sima-zhu commented 2 years ago

The scope of the issue: