Team Epic: Lab Reliability, Accountability and Telemetry

[trylek]

This epic covers developer interaction with lab pipelines (AzDO); provisions for being able to analyze and investigate AzDO failures and other related forms of telemetry.

• Lab stability and performance
• Reproduction of Helix environment
• Diagnostic files produced by CI
• Dump management
• Documentation

Lab stability and performance (5 weeks total)

• [ ] (2 days, P3) Remove x64 binaries before submitting crossgen comparison job; https://github.com/dotnet/runtime/issues/34103
• [ ] (2 weeks, P2) Fix IL symbols in .NET Core 2.1; https://github.com/dotnet/runtime/issues/33550
• [ ] (2 weeks, P2) Protect Ubuntu by using registry-based image to run in CI; https://github.com/dotnet/runtime/issues/32309
• [ ] (1 week to design, P2) Add more specific filtering for ARM / ARM64 testing; https://github.com/dotnet/runtime/issues/33070

Reproduction of Helix environment (13 weeks total)

• [ ] (3 weeks, P1) Making it easier to retrieve repro files for a Helix work item; https://github.com/dotnet/runtime/issues/45700
• [ ] (4 weeks, P1) Means to construct a repro environment (e.g. a local Hyper-V, an Azure VM) based on the Helix configuration; https://github.com/dotnet/runtime/issues/45701
• [ ] (2 weeks, P1) Support for running Helix client execution scripts in the repro environment; https://github.com/dotnet/runtime/issues/45702
• [ ] (4 weeks, P1) Injecting debugging / intervention points in the work item execution pipeline; https://github.com/dotnet/runtime/issues/45704

Diagnostic files produced by CI (7 weeks total)

• [ ] (2 weeks, P1) Produce msbuild binlog files for all build phases that today produce the triplets of logs / errors / warnings; https://github.com/dotnet/runtime/issues/45706
• [ ] (3 weeks, P1) Produce dumps for crashing build tools on the AzDO build machines; https://github.com/dotnet/runtime/issues/45708
• [x] (2 weeks to investigate, P2) Make sure that all Helix test failures including timeouts are covered by dumps; https://github.com/dotnet/runtime/issues/45709

Dump management (5 weeks total)

• [ ] (1 week, P1) Improve documentation on retrieving and using dumps; https://github.com/dotnet/runtime/issues/45711
• [ ] (4 weeks, P2) Investigate means for automated dump & symbol download followed by something like SOS !analyze, ideally made part of logs for the failed test; https://github.com/dotnet/runtime/issues/45712

Documentation

[danmoseley]

@jeffschwMSFT is it OK to remove Team Epic from the various titles ? I think it's a bit confusing

[jeffschwMSFT]

I am neutral, what is confusing about the title?

[trylek]

I think that Dan's point was just that having "Team Epic" at the front of each title seems a bit superfluous.

[jeffschwMSFT]

I am fine either way. I feel the titles are unique to the issue at hand, so I was curious if this was hitting a query or something.