Managed PGO auditing

[AndyAyersMS]

Now that we have static managed PGO data flowing around, we need a way to track and audit the data:

• updates to the managed PGO data should be visible as commits in the repo, so we can reproduce old build behaviors and have a marker point in the perf history for when updates happen
  • also would be useful for perf investigations to be able to mix and match data (say using newer data in an old build)
• we should be able to determine the build that was used to gather the PGO data, and (perhaps) audit changes made to the collection process
  • in particular the time lag between collection and consumption is interesting. If this value gets too large we might need some way to raise priority of investigation on why there is no new data
• it would be good to develop tooling that can compare sets of PGO data to describe in some manner how similar/different the data is, and generate and make those delta summaries available
• it would be good to have some characterization of how much PGO data gets rejected in crossgen due to versioning lag (native PGO tools produce similar reports)
• Perf lab runs should do some non-PGO runs alongside the now default PGO runs (say via COMPlus_JitDisablePgo=1), so we can
  • keep tabs on the overall impact of static PGO
  • more easily spot when a perf change is solely due to a change in static PGO data

cc @davidwrighton @DrewScoggins

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[davidwrighton]

@AndyAyersMS updates the the managed pgo data are handled by dependency flow, so they are already visible as commits. Updates are visible as changes to eng/Versions.props and in particular when optimizationwindows_ntx64MIBCRuntimeVersion or the other related variables change. However, the rest of your points are quite interesting.

[AndyAyersMS]

@davidwrighton thanks. So I take it we have not had an update since the initial checkin, and that data was gathered on 3/17?

https://github.com/dotnet/runtime/blame/082af357d39f9df590fb4b123bcb6c21b0dd4369/eng/Versions.props#L120-L122

Have you any ideas on a reasonable cadence for updates?

@DrewScoggins another possibility for what we were seeing in some of those perf tests is that the FX code (at the IL level) has shifted away from what it was back on 3/17 or thereabouts, and so PGO data no longer applies. Currently we can only tell this is happening indirectly.

In general if we don't update PGO data we expect to see a decline in perf over time as mismatches between what was measured and what is being prejitted or jitted become more and more pronounced. For any given test the jumps may be at particular points; if we could "integrate" all these over a lot of test we'd see more of a smooth overall degradation.

[davidwrighton]

Correct, we haven't had an update since initial checkin. @Lxiamail is working on the build so that we'll have regular flow, but the build infrastructure has proven difficult to update. Once that is up and running it should be straightforward to receive updates about once a day.