search

briancylui / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
http://dot.net/ml
MIT License
0 stars 0 forks source link

Stabilized perf results for KMeansAndLogisticRegression benchmark #10

Open briancylui opened 6 years ago

briancylui commented 6 years ago 
// * Legends *
  Mean         : Arithmetic mean of all measurements
  Error        : Half of 99.9% confidence interval
  StdDev       : Standard deviation of all measurements
  Extra Metric : Value of the provided extra metric
  Gen 0        : GC Generation 0 collects per 1k Operations
  Gen 1        : GC Generation 1 collects per 1k Operations
  Gen 2        : GC Generation 2 collects per 1k Operations
  Allocated    : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 s          : 1 Second (1 sec)

netcoreapp3.0 with my change:

Iteration 1:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-YPIYKX : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 1.781 s 0.0976 s 0.1044 s - 16000.0000 8000.0000 3000.0000 47.57 MB

Iteration 2:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Max: 1.90GHz) (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-GREASD : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 1.781 s 0.1050 s 0.1209 s - 16000.0000 8000.0000 3000.0000 47.27 MB

Iteration 3:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-OHSMOJ : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 1.853 s 0.0926 s 0.1067 s - 16000.0000 8000.0000 3000.0000 47.17 MB

Iteration 4:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-MIAXEM : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 1.797 s 0.1104 s 0.1271 s - 16000.0000 8000.0000 3000.0000 47.27 MB

netcoreapp3.0 without my change:

Iteration 1:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-TIGZKX : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 2.094 s 0.0440 s 0.0489 s - 17000.0000 8000.0000 3000.0000 47.13 MB

Iteration 2:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Max: 1.90GHz) (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-LMZGNT : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 2.062 s 0.0828 s 0.0920 s - 17000.0000 8000.0000 3000.0000 47.17 MB

Iteration 3:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-NAKODA : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 2.139 s 0.0677 s 0.0780 s - 17000.0000 9000.0000 3000.0000 47.12 MB

Iteration 4:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-QVPJCB : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 2.079 s 0.0778 s 0.0896 s - 17000.0000 9000.0000 3000.0000 47.36 MB
briancylui commented 6 years ago

@danmosemsft @eerhardt @safern @tannergooding: In netcoreapp3.0, perf results with my change are at least 10% better than those without my change. Will present these.

eerhardt commented 6 years ago

This is awesome results, @briancylui! I'm excited to see the presentation tomorrow.

safern commented 6 years ago

This is great!! I love the fact that ML.NET can be used in different architectures, by building on one of them and getting perf benefits!

helloguo commented 6 years ago

@briancylui can you share how to build and run ML.NET and Benchmark KMeansAndLogisticRegression with .NET Core 3.0 with your change?