Testing - Optimize regression testing performance

[SubPointSupport]

Currently, the full regression testing takes a lot of time to complete.

• Full SSOM run takes ~3-4+ hours
• Full CSOM run takes ~3-4+ hours
• Full O365 run takes around 9+ hours

Also, O365 can easily hit daily CSOM query limits. In that case, parallel execution and multiple site collection are to be considered.

Initially, NCrunch should be used to spread regression testing over 8-12 SharePoint farms. Automated Hyper-V, Vagrant + VirtualBox, Azure/AWS based environments are to be used. In addition, some optimizations in regression tests can be introduced (better CSOM queries, reducing isolation to limit the amount of the new webs created).

• [ ] The bottleneck should be identified and eliminated
• [x] Better usage of NCrunch should be considered
• [x] Offloading tests run to Azure VMs via NCrunch should be considered

NCrunch usage grid with 6-8-12 SharePoint 2013 farms reduces the total regression testing times to 20-35 minutes. That's 6-8-10 times faster that regular sequential test execution on a single farm. Same works well for O365 testing with randomized site collections and 4-6 grid nodes for NCrunch.

[SubPointSupport]

Related to Testing - Implement "Regression.Performance" test coverage #232