Increase VM efficiency

[markwilkie]

As managing our budget becomes more pressing, we have an increased business need to manage our Azure budgets well. The scaling epic is part of this effort, but there is other work to be done. e.g. spot VMs.

• [x] Execute servicing builds and tests to their own subscription under COGS
• [ ] Leverage Spot VM's in our test queues to reduce cost
• [ ] Investigate and create proposal for using cheaper VM's when waiting for Helix to complete
• [x] Investigate and create proposal for moving build queues into another hosted solution
  • AzDo scaleset pools
  • 1es hosted pools
  • Helix hosted pools

    Recently Triaged Issues

All issues in this section should be triaged by the v-team into one of their business objectives or features.

• [x] https://github.com/dotnet/core-eng/issues/7980
• [x] https://github.com/dotnet/core-eng/issues/9885
• [x] https://github.com/dotnet/core-eng/issues/9944
• [x] https://github.com/dotnet/core-eng/issues/10864
• [x] https://github.com/dotnet/core-eng/issues/11107
• [x] https://github.com/dotnet/core-eng/issues/11409

[danmoseley]

Adding link since I didn't know what Spot VM's are - they are non guaranteed VM's that may not be available or may get reclaimed ('evicted') suddenly, but are 50-80% cheaper https://docs.microsoft.com/en-us/azure/virtual-machines/spot-vms