search

dotnet / jitutils

MIT License
146 stars 59 forks source link

Proposal: PMI asm diffs using SuperPMI #185

Open BruceForstall opened 5 years ago

BruceForstall commented 5 years ago

We now have jit-diff diff --pmi to use PMI for generating assembly code across a specified corpus of code, and analyzing the diffs assuming both baseline and diff compiler asm code is generated. This proposal is to introduce a mode to user SuperPMI to achieve the same results, but hopefully faster, eliminating incorrect diffs, and allowing for easier re-JIT of interesting methods.

I propose we add a --superpmi option, such that you would use jit-diff diff --pmi --superpmi. This specifies to still use PMI, but to accelerate it using SuperPMI. In this mode, you would be required to specify both a baseline and diff compiler (SuperPMI asm diffs require both). These steps would be executed:

The resulting CODE.MCH and DIFF.MCL can remain for subsequent re-use and further investigations.

Note that we could also have a jit-diff diff --superpmi (without --pmi) for doing a similar thing but for crossgen-based asm diffs.

@dotnet/jit-contrib

CarolEidt commented 5 years ago

Sounds great to me!

jashook commented 5 years ago

I think that this can be improved to share the resulting mch files more broadly. I also believe that it can be can be generalized such that pmi is not the only aspect of the collection to run diffs on.

I can instead image a workflow like:

CI Workflow

  1. Run SuperPMI collection (xarch)
    • superpmi collect - runtest.sh x64 checked
    • superpmi collect - pmi assemblies
    • superpmi collect - jitbench
  2. Public resulting .mchs to* :

Dev Workflow

  1. Run jit-diff (or some tool) --superpmi arch build_type base jit diff jit
    • Pull down the latest-collection.mch
    • Run SuperPMI in asm diff mode, passing both the baseline and diff JIT, and the CODE.MCH file, generating a DIFF.MCL file identifying all the functions that differ.
    • If there are any diffs, generate textual ASM output (using COMPlus_JitDisasm) by running SuperPMI over just the DIFF.MCL methods, once for the baseline JIT, and once for the diff JIT.
    • Run jit-analyze over the resulting ASM, to generate the diff statistics.

*(hypothetical links) they do not go anywhere.

jashook commented 5 years ago

Note that either proposal requires coredistools to be ported to:

  1. linux arm64
  2. linux arm32
  3. OSX
  4. Windows arm32
  5. Windows arm64
BruceForstall commented 5 years ago

@jashook These two proposals can be complementary. I would like to have the CI generate and store superpmi collections across multiple platforms/architectures, and have the dev workflow pull down the appropriate one for use in superpmi replay for assert checking or diffs. That proposal involves more moving parts, and working with the CI. My proposal would probably require building basic tooling in jitutils that could be leveraged for both cases, and only depends on the local machine, with no CI interaction. So hopefully it would be quicker to build and more robust.

Perhaps a jitutils tool that should come out of this is a "jit-spmi collect" tool that would collect whatever you want and put it in some file -- COLLECTION.MCH, say. Then "jit-spmi replay" could do assert checking using this, and "jit-spmi diff" or "jit-diff diff --superpmi" or something would use it for asm diffs.

AndyAyersMS commented 4 years ago

Some current, useful jit-diff functionality that might require extra work if we leverage SuperPMI:

BruceForstall commented 4 years ago

Reporting metric changes as a percentage of the metric for all methods.

I've thought we needed to add that capability to SuperPMI asm diffs for a long time. For something like code size, where SuperPMI can itself compute the metric, it seems not too difficult. For metrics where only the JIT can compute the metric, we perhaps to teach the JIT how to report any/all metrics back to the runtime through the the JIT/EE interface, which would allow SuperPMI to capture them for subsequent use.