"Live Only" test failures should be identified and flagged with high priority

[tjprescott]

Is your feature request related to a problem? Please describe. We have a number of tests that are marked @live_only or LiveScenarioTest because they cannot be replayed as a recording. In these cases, the nightly live run is the only way to detect a CLI regression. Unfortunately, due to the noisiness of the live test system, regressions are not identified and corrected immediately.

Describe the solution you'd like All data from the live test runs are stored in a database. We can analyze our tests to identify exactly which ones are live only. We should be able to check the daily live test failures and compare it against this list. If a live-only test fails, it should trigger a more elevated response so that it gets immediate attention.

Describe alternatives you've considered Aside from an automated solution, this scenario relies on consistent, dedicated human intervention, which is not realistic.

[yonzhan]

@bim-msft add to S162.

[yungezz]

for live test, what we did so far

• setup live test pipeline, so that live test could be run regularly or adhoc.
• internal doc on how to use live test during sdk version bumping testing scenario

To do

• refine email notification for test result
• enable regular live test.

future we'll investigate how to store live test result into data store for longer time data mining. Will close the issue after the To do part done.