elizalcodes
commented
9 months ago running
crank compare master.json branch.json
fixed the issue. Which makes sense as we want to compare branch against master results.
it would be nice to be able to compare them in any order, but this solved my issue.
Running crank compare branch.json master.json hits the following exception, when branch.json contains one more benchmark result than master.json
System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index') at System.Collections.Generic.List1.getItem(Int32 index) at Microsoft.Crank.Controller.ResultTable.CalculateColumnWidths() in //src/Microsoft.Crank.Controller/ResultTable.cs:line 48 at Microsoft.Crank.Controller.ResultTable.Render(TextWriter writer) in //src/Microsoft.Crank.Controller/ResultTable.cs:line 67 at Microsoft.Crank.Controller.ResultComparer.DisplayDiff(IEnumerable1 allBenchmarks, IEnumerable1 allNames) in //src/Microsoft.Crank.Controller/ResultComparer.cs:line 314 at Microsoft.Crank.Controller.ResultComparer.Compare(IEnumerable1 filenames, JobResults jobResults, Benchmark[] benchmarks, String jobName) in /_/src/Microsoft.Crank.Controller/ResultComparer.cs:line 64 at Microsoft.Crank.Controller.Program.<>c__DisplayClass53_1.<Main>b__3() in /_/src/Microsoft.Crank.Controller/Program.cs:line 226 at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass144_0.<OnExecute>b__0(CancellationToken _) at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken) at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args) at Microsoft.Crank.Controller.Program.Main(String[] args) in /_/src/Microsoft.Crank.Controller/Program.cs:line 692branch.json master.json
notice that there are 7 columns for each benchmark row in the table. For some reason the new benchmark has only 3 columns, while table builder expects 7 and tries to iterate 7 times.