search

dotnet / jitutils

MIT License
144 stars 59 forks source link

Add --concat-files option to jit-analyze #393

Closed jakobbotsch closed 6 months ago

jakobbotsch commented 7 months ago

Specifying this option makes jit-analyze concatenate all files in the input base/diff directories and treat them as a single logical unit.

Example: To produce all .dasm files for a change (instead of just some hand picked example ones), one can pass -metrics CodeSize to superpmi.py: superpmi.py asmdiffs -metrics CodeSize

This creates base/diff directories with a single .dasm file for every code size difference. Running jit-analyze --base <path to base dir> --diff <path to diff dir> gives

Found 3820 files with textual diffs.

Summary of Code Size diffs:
(Lower is better)

Total bytes of base: 4017093
Total bytes of diff: 4067605
Total bytes of delta: 50512 (1.26 % of base)
Total relative delta: 72.45
    diff is a regression.
    relative diff is a regression.

Top file regressions (bytes):
         198 : 64336.dasm (17.82% of base)
         184 : 68141.dasm (16.28% of base)
         182 : 70971.dasm (17.15% of base)
         181 : 65842.dasm (17.08% of base)
         133 : 7970.dasm (9.40% of base)
         121 : 107773.dasm (2.92% of base)
         115 : 97615.dasm (2.77% of base)
         106 : 89159.dasm (2.30% of base)
         106 : 32005.dasm (2.25% of base)
         106 : 89036.dasm (2.25% of base)
          96 : 84331.dasm (3.34% of base)
          81 : 107921.dasm (14.14% of base)
          81 : 29722.dasm (14.14% of base)
          81 : 106543.dasm (14.14% of base)
          81 : 107901.dasm (14.14% of base)
          81 : 105724.dasm (14.14% of base)
          81 : 106853.dasm (14.14% of base)
          77 : 107700.dasm (2.58% of base)
          73 : 56995.dasm (12.56% of base)
          73 : 34037.dasm (12.56% of base)

3820 total files with Code Size differences (0 improved, 3820 regressed), 0 unchanged.

Top method regressions (bytes):
         198 (17.82% of base) : 64336.dasm - Algorithms.VectorFloatRenderer:RenderSingleThreadedNoADT(float,float,float,float,float):this (Tier1-OSR)
         184 (16.28% of base) : 68141.dasm - Algorithms.VectorDoubleRenderer:RenderSingleThreadedNoADT(float,float,float,float,float):this (Tier1-OSR)
         182 (17.15% of base) : 70971.dasm - Algorithms.VectorDoubleRenderer:RenderSingleThreadedWithADT(float,float,float,float,float):this (Tier1-OSR)
         181 (17.08% of base) : 65842.dasm - Algorithms.VectorFloatRenderer:RenderSingleThreadedWithADT(float,float,float,float,float):this (Tier1-OSR)
         133 ( 9.40% of base) : 7970.dasm - System.RuntimeType:GetConstructorCandidates(System.String,int,int,System.Type[],ubyte):System.RuntimeType+ListBuilder`1[System.Reflection.ConstructorInfo]:this (Tier1)
         121 ( 2.92% of base) : 107773.dasm - System.Text.Unicode.Utf8Utility:TranscodeToUtf8(ulong,int,ulong,int,byref,byref):int (Tier1)
         115 ( 2.77% of base) : 97615.dasm - System.Text.Unicode.Utf8Utility:TranscodeToUtf8(ulong,int,ulong,int,byref,byref):int (Tier1)
         106 ( 2.30% of base) : 89159.dasm - System.Text.RegularExpressions.CompiledRegexRunner:Regex1_TryMatchAtCurrentPosition(System.Text.RegularExpressions.RegexRunner,System.ReadOnlySpan`1[ushort]):ubyte (FullOpts)
         106 ( 2.25% of base) : 32005.dasm - System.Text.RegularExpressions.CompiledRegexRunner:Regex1_TryMatchAtCurrentPosition(System.Text.RegularExpressions.RegexRunner,System.ReadOnlySpan`1[ushort]):ubyte (FullOpts)
         106 ( 2.25% of base) : 89036.dasm - System.Text.RegularExpressions.CompiledRegexRunner:Regex1_TryMatchAtCurrentPosition(System.Text.RegularExpressions.RegexRunner,System.ReadOnlySpan`1[ushort]):ubyte (FullOpts)
          96 ( 3.34% of base) : 84331.dasm - Benchstone.BenchI.Puzzle:DoIt():ubyte:this (Tier1)
          81 (14.14% of base) : 107921.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 29722.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 106543.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 107901.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 105724.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 106853.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          77 ( 2.58% of base) : 107700.dasm - System.Text.Unicode.Utf8Utility:TranscodeToUtf16(ulong,int,ulong,int,byref,byref):int (Tier1)
          73 (12.56% of base) : 56995.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          73 (12.56% of base) : 34037.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)

Top method regressions (percentages):
         198 (17.82% of base) : 64336.dasm - Algorithms.VectorFloatRenderer:RenderSingleThreadedNoADT(float,float,float,float,float):this (Tier1-OSR)
         182 (17.15% of base) : 70971.dasm - Algorithms.VectorDoubleRenderer:RenderSingleThreadedWithADT(float,float,float,float,float):this (Tier1-OSR)
         181 (17.08% of base) : 65842.dasm - Algorithms.VectorFloatRenderer:RenderSingleThreadedWithADT(float,float,float,float,float):this (Tier1-OSR)
         184 (16.28% of base) : 68141.dasm - Algorithms.VectorDoubleRenderer:RenderSingleThreadedNoADT(float,float,float,float,float):this (Tier1-OSR)
          81 (14.14% of base) : 107921.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 29722.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 106543.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 107901.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 105724.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          81 (14.14% of base) : 106853.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          24 (14.12% of base) : 101959.dasm - System.Runtime.CompilerServices.CastHelpers:IsInstanceOfAny(ulong,System.Object):System.Object (Tier1)
          11 (13.25% of base) : 91041.dasm - System.Collections.Generic.GenericArraySortHelper`1[System.Collections.IntStruct]:InsertionSort(System.Span`1[System.Collections.IntStruct]) (Tier1)
          11 (13.25% of base) : 85527.dasm - System.Collections.Generic.GenericArraySortHelper`1[System.Collections.IntStruct]:InsertionSort(System.Span`1[System.Collections.IntStruct]) (Tier1)
          57 (12.58% of base) : 85279.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          57 (12.58% of base) : 93696.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          57 (12.58% of base) : 62080.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          57 (12.58% of base) : 93491.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          57 (12.58% of base) : 101375.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          57 (12.58% of base) : 105283.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)
          57 (12.58% of base) : 88622.dasm - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1)

3820 total methods with Code Size differences (0 improved, 3820 regressed), 0 unchanged.

If you pass --concat-files, you get:

Found 3820 files with textual diffs.

Summary of Code Size diffs:
(Lower is better)

Total bytes of base: 4017093
Total bytes of diff: 4067605
Total bytes of delta: 50512 (1.26 % of base)
Total relative delta: 3.26
    diff is a regression.
    relative diff is a regression.

Top file regressions (bytes):
       50512 : diff (1.26% of base)

1 total files with Code Size differences (0 improved, 1 regressed), 0 unchanged.

Top method regressions (bytes):
       12940 ( 1.27% of base) : base - System.Text.Ascii:NarrowUtf16ToAscii(ulong,ulong,ulong):ulong (Tier1) (630 methods)
       10990 (10.48% of base) : base - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1) (216 methods)
        3546 ( 0.84% of base) : base - System.Text.Unicode.Utf8Utility:TranscodeToUtf8(ulong,int,ulong,int,byref,byref):int (Tier1) (109 methods)
        3156 ( 1.29% of base) : base - System.SpanHelpers:IndexOfNullCharacter(ulong):int (Tier1) (450 methods)
        2512 ( 0.42% of base) : base - System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[System.__Canon]:PopulateProperties(System.RuntimeType+RuntimeTypeCache+Filter,System.RuntimeType,System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.List`1[System.Reflection.RuntimePropertyInfo]],System.Span`1[ubyte],ubyte,byref):this (Tier1) (155 methods)
        1757 ( 3.17% of base) : base - System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[System.__Canon]:MergeWithGlobalList(System.__Canon[]):this (Tier1) (84 methods)
        1391 ( 0.88% of base) : base - System.RuntimeType:GetPropertyCandidates(System.String,int,System.Type[],ubyte):System.RuntimeType+ListBuilder`1[System.Reflection.PropertyInfo]:this (Tier1) (134 methods)
        1349 ( 5.31% of base) : base - System.Runtime.CompilerServices.CastHelpers:ChkCastAny(ulong,System.Object):System.Object (Tier1) (129 methods)
        1335 ( 0.40% of base) : base - System.RuntimeType:FilterApplyMethodBase(System.Reflection.MethodBase,int,int,int,System.Type[]):ubyte (Tier1) (473 methods)
        1274 ( 5.06% of base) : base - System.Runtime.CompilerServices.CastHelpers:IsInstanceOfAny(ulong,System.Object):System.Object (Tier1) (147 methods)
        1120 ( 0.78% of base) : base - System.Text.Unicode.Utf8Utility:TranscodeToUtf16(ulong,int,ulong,int,byref,byref):int (Tier1) (50 methods)
         633 ( 0.87% of base) : base - System.Text.Ascii:GetIndexOfFirstNonAsciiChar_Vector(ulong,ulong):ulong (Tier1) (78 methods)
         561 ( 1.59% of base) : base - System.Collections.Generic.Dictionary`2[System.__Canon,System.__Canon]:FindValue(System.__Canon):byref:this (Tier1) (78 methods)
         530 ( 2.40% of base) : base - System.Number:TryParseBinaryIntegerStyle[ushort,int](System.ReadOnlySpan`1[ushort],int,System.Globalization.NumberFormatInfo,byref):int (Tier1) (20 methods)
         522 ( 0.82% of base) : base - System.RuntimeType:MakeGenericType(System.Type[]):System.Type:this (Tier1) (35 methods)
         452 ( 1.11% of base) : base - System.Collections.Generic.Dictionary`2[System.__Canon,System.__Canon]:TryGetValue(System.__Canon,byref):ubyte:this (Tier1) (80 methods)
         423 ( 0.59% of base) : base - System.Collections.HashHelpers:GetPrime(int):int (Tier1) (423 methods)
         321 ( 2.10% of base) : base - System.Text.RegularExpressions.CompiledRegexRunner:Regex1_TryMatchAtCurrentPosition(System.Text.RegularExpressions.RegexRunner,System.ReadOnlySpan`1[ushort]):ubyte (FullOpts) (4 methods)
         251 ( 2.43% of base) : base - System.Text.Unicode.Utf8Utility:TranscodeToUtf8(ulong,int,ulong,int,byref,byref):int (Tier1-OSR) (5 methods)
         239 ( 3.61% of base) : base - System.Collections.Generic.HashSet`1[System.__Canon]:FindItemIndex(System.__Canon):int:this (Tier1) (14 methods)

Top method regressions (percentages):
         182 (17.15% of base) : base - Algorithms.VectorDoubleRenderer:RenderSingleThreadedWithADT(float,float,float,float,float):this (Tier1-OSR)
         181 (17.08% of base) : base - Algorithms.VectorFloatRenderer:RenderSingleThreadedWithADT(float,float,float,float,float):this (Tier1-OSR)
         184 (16.28% of base) : base - Algorithms.VectorDoubleRenderer:RenderSingleThreadedNoADT(float,float,float,float,float):this (Tier1-OSR)
          22 (13.25% of base) : base - System.Collections.Generic.GenericArraySortHelper`1[System.Collections.IntStruct]:InsertionSort(System.Span`1[System.Collections.IntStruct]) (Tier1) (2 methods)
       10990 (10.48% of base) : base - System.SpanHelpers:IndexOfNullByte(ulong):int (Tier1) (216 methods)
          22 ( 9.09% of base) : base - System.Collections.Generic.GenericArraySortHelper`1[System.Collections.BigStruct]:InsertionSort(System.Span`1[System.Collections.BigStruct]) (Tier1) (2 methods)
         199 ( 8.96% of base) : base - Algorithms.VectorFloatRenderer:RenderSingleThreadedNoADT(float,float,float,float,float):this (Tier1-OSR) (2 methods)
          35 ( 7.92% of base) : base - IDEAEncryption:DoIDEAIteration(ubyte[],ubyte[],ubyte[],int,int,ushort[],ushort[]):long (Tier1-OSR)
          11 ( 5.82% of base) : base - BenchmarksGame.KNucleotide_9:loadThreeData(System.IO.Stream) (Tier1-OSR)
          30 ( 5.32% of base) : base - Benchstone.BenchF.Bisect:Test():ubyte:this (Tier1-OSR)
        1349 ( 5.31% of base) : base - System.Runtime.CompilerServices.CastHelpers:ChkCastAny(ulong,System.Object):System.Object (Tier1) (129 methods)
          22 ( 5.19% of base) : base - System.Buffers.SharedArrayPoolPartitions:TryPop():System.Array:this (Tier1) (2 methods)
          29 ( 5.17% of base) : base - System.Buffers.Text.Base64:DecodeFromUtf8InPlace(System.Span`1[ubyte],byref,ubyte):int (Tier1-OSR)
        1274 ( 5.06% of base) : base - System.Runtime.CompilerServices.CastHelpers:IsInstanceOfAny(ulong,System.Object):System.Object (Tier1) (147 methods)
          33 ( 4.81% of base) : base - BenchmarksGame.FannkuchRedux_5:run(int,int,int) (Tier1-OSR)
          44 ( 4.59% of base) : base - LUDecomp:DoLUIteration(double[][],double[],double[][][],double[][],int):long (Tier1-OSR) (2 methods)
          75 ( 4.08% of base) : base - System.Runtime.Serialization.SerializationInfo:ThrowIfDeserializationInProgress(System.String,byref) (Tier1) (3 methods)
         156 ( 3.94% of base) : base - System.Text.RegularExpressions.Tests.Perf_Regex_Cache:CreatePatterns(int,int):System.String[] (Tier1-OSR) (12 methods)
         239 ( 3.61% of base) : base - System.Collections.Generic.HashSet`1[System.__Canon]:FindItemIndex(System.__Canon):int:this (Tier1) (14 methods)
         118 ( 3.56% of base) : base - System.IO.Tests.Perf_RandomAccess:Write(long,int,int):this (Tier1-OSR) (2 methods)

183 total methods with Code Size differences (0 improved, 183 regressed), 0 unchanged.

It would be even nicer if superpmi.py asmdiffs could do this automatically, but sadly we do not have a cheap way to get access to the method name of a context without producing its .dasm file, and that's very slow to do for all contexts with diffs in them for large changes.

cc @dotnet/jit-contrib