search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.41k stars 957 forks source link

R plots: issue with dplyr #692

Closed lol768 closed 6 years ago

lol768 commented 6 years ago

I am unable to generate plots, the images do not get written into the results directory.

Supporting information:

SystemPerformanceTests.log 
// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   SystemPerformanceTests.TestIntegrationWithResourceBounds: Core(Runtime=Core)

// Validating benchmarks:
Assembly GranularPermissions which defines benchmarks references non-optimized Loyc.Essentials
// ***** Building 1 benchmark(s) in Parallel: Start   *****
// ***** Done, took 00:00:06 (6.33 sec)   *****
// **************************
// Benchmark: SystemPerformanceTests.TestIntegrationWithResourceBounds: Core(Runtime=Core)
// *** Execute ***
// Launch: 1 / 1
Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied
// BeforeAnythingElse

// Benchmark Process Environment Information:
// Runtime=.NET Core 2.0.5 (CoreCLR 4.6.0.0, CoreFX 4.6.26018.01), 64bit RyuJIT
// GC=Concurrent Workstation
// Job: Core(Runtime=Core)

Pilot  1: 16 op, 770841.00 ns, 48.1776 us/op
Pilot  2: 32 op, 964264.00 ns, 30.1333 us/op
Pilot  3: 64 op, 1803485.00 ns, 28.1795 us/op
Pilot  4: 128 op, 3887595.00 ns, 30.3718 us/op
Pilot  5: 256 op, 8068376.00 ns, 31.5171 us/op
Pilot  6: 512 op, 15818801.00 ns, 30.8961 us/op
Pilot  7: 1024 op, 32995329.00 ns, 32.2220 us/op
Pilot  8: 2048 op, 70416953.00 ns, 34.3833 us/op
Pilot  9: 4096 op, 137492239.00 ns, 33.5674 us/op
Pilot 10: 8192 op, 294344640.00 ns, 35.9307 us/op
Pilot 11: 16384 op, 579518765.00 ns, 35.3710 us/op

IdleWarmup  1: 16384 op, 44355.00 ns, 2.7072 ns/op
IdleWarmup  2: 16384 op, 34787.00 ns, 2.1232 ns/op
IdleWarmup  3: 16384 op, 33726.00 ns, 2.0585 ns/op
IdleWarmup  4: 16384 op, 34845.00 ns, 2.1268 ns/op
IdleWarmup  5: 16384 op, 33729.00 ns, 2.0587 ns/op
IdleWarmup  6: 16384 op, 33771.00 ns, 2.0612 ns/op
IdleWarmup  7: 16384 op, 33803.00 ns, 2.0632 ns/op
IdleWarmup  8: 16384 op, 34855.00 ns, 2.1274 ns/op
IdleWarmup  9: 16384 op, 33798.00 ns, 2.0629 ns/op

IdleTarget  1: 16384 op, 33763.00 ns, 2.0607 ns/op
IdleTarget  2: 16384 op, 34868.00 ns, 2.1282 ns/op
IdleTarget  3: 16384 op, 34890.00 ns, 2.1295 ns/op
IdleTarget  4: 16384 op, 34804.00 ns, 2.1243 ns/op
IdleTarget  5: 16384 op, 33854.00 ns, 2.0663 ns/op
IdleTarget  6: 16384 op, 48519.00 ns, 2.9614 ns/op
IdleTarget  7: 16384 op, 34782.00 ns, 2.1229 ns/op
IdleTarget  8: 16384 op, 33797.00 ns, 2.0628 ns/op
IdleTarget  9: 16384 op, 34869.00 ns, 2.1282 ns/op
IdleTarget 10: 16384 op, 34871.00 ns, 2.1284 ns/op
IdleTarget 11: 16384 op, 34872.00 ns, 2.1284 ns/op
IdleTarget 12: 16384 op, 34858.00 ns, 2.1276 ns/op
IdleTarget 13: 16384 op, 34778.00 ns, 2.1227 ns/op
IdleTarget 14: 16384 op, 34989.00 ns, 2.1356 ns/op
IdleTarget 15: 16384 op, 34868.00 ns, 2.1282 ns/op

MainWarmup  1: 16384 op, 581248368.00 ns, 35.4766 us/op
MainWarmup  2: 16384 op, 564911350.00 ns, 34.4795 us/op
MainWarmup  3: 16384 op, 574649786.00 ns, 35.0738 us/op
MainWarmup  4: 16384 op, 580201283.00 ns, 35.4127 us/op
MainWarmup  5: 16384 op, 584931333.00 ns, 35.7014 us/op
MainWarmup  6: 16384 op, 574011693.00 ns, 35.0349 us/op
MainWarmup  7: 16384 op, 586989677.00 ns, 35.8270 us/op
MainWarmup  8: 16384 op, 578507003.00 ns, 35.3093 us/op

// BeforeMainRun
MainTarget  1: 16384 op, 596989511.00 ns, 36.4373 us/op
MainTarget  2: 16384 op, 594631805.00 ns, 36.2934 us/op
MainTarget  3: 16384 op, 599310538.00 ns, 36.5790 us/op
MainTarget  4: 16384 op, 587569066.00 ns, 35.8624 us/op
MainTarget  5: 16384 op, 597122460.00 ns, 36.4455 us/op
MainTarget  6: 16384 op, 581235851.00 ns, 35.4758 us/op
MainTarget  7: 16384 op, 595930371.00 ns, 36.3727 us/op
MainTarget  8: 16384 op, 574032883.00 ns, 35.0362 us/op
MainTarget  9: 16384 op, 588430790.00 ns, 35.9150 us/op
MainTarget 10: 16384 op, 579486975.00 ns, 35.3691 us/op
MainTarget 11: 16384 op, 584176039.00 ns, 35.6553 us/op
MainTarget 12: 16384 op, 576776648.00 ns, 35.2037 us/op
MainTarget 13: 16384 op, 592326744.00 ns, 36.1528 us/op
MainTarget 14: 16384 op, 578377139.00 ns, 35.3013 us/op
MainTarget 15: 16384 op, 581454503.00 ns, 35.4892 us/op

// AfterMainRun
Result  1: 16384 op, 596953952.20 ns, 36.4352 us/op
Result  2: 16384 op, 594596246.20 ns, 36.2913 us/op
Result  3: 16384 op, 599274979.20 ns, 36.5768 us/op
Result  4: 16384 op, 587533507.20 ns, 35.8602 us/op
Result  5: 16384 op, 597086901.20 ns, 36.4433 us/op
Result  6: 16384 op, 581200292.20 ns, 35.4737 us/op
Result  7: 16384 op, 595894812.20 ns, 36.3705 us/op
Result  8: 16384 op, 573997324.20 ns, 35.0340 us/op
Result  9: 16384 op, 588395231.20 ns, 35.9128 us/op
Result 10: 16384 op, 579451416.20 ns, 35.3669 us/op
Result 11: 16384 op, 584140480.20 ns, 35.6531 us/op
Result 12: 16384 op, 576741089.20 ns, 35.2015 us/op
Result 13: 16384 op, 592291185.20 ns, 36.1506 us/op
Result 14: 16384 op, 578341580.20 ns, 35.2992 us/op
Result 15: 16384 op, 581418944.20 ns, 35.4870 us/op
GC:  0 0 0 0 0

// AfterAll

Mean = 35.8371 us, StdErr = 0.1330 us (0.37%); N = 15, StdDev = 0.5152 us
Min = 35.0340 us, Q1 = 35.3669 us, Median = 35.8602 us, Q3 = 36.3705 us, Max = 36.5768 us
IQR = 1.0036 us, LowerFence = 33.8615 us, UpperFence = 37.8760 us
ConfidenceInterval = [35.2863 us; 36.3878 us] (CI 99.9%), Margin = 0.5508 us (1.54% of Mean)
Skewness = -0.01, Kurtosis = 1.36

// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts/results/SystemPerformanceTests-measurements.csv
  BuildPlots.R
  BenchmarkDotNet.Artifacts/results/SystemPerformanceTests-boxplot.png
  BenchmarkDotNet.Artifacts/results/SystemPerformanceTests-barplot.png
  BenchmarkDotNet.Artifacts/results/SystemPerformanceTests-report.csv
  BenchmarkDotNet.Artifacts/results/SystemPerformanceTests-report-github.md
  BenchmarkDotNet.Artifacts/results/SystemPerformanceTests-report.html

// * Detailed results *
SystemPerformanceTests.TestIntegrationWithResourceBounds: Core(Runtime=Core)
Runtime = .NET Core 2.0.5 (CoreCLR 4.6.0.0, CoreFX 4.6.26018.01), 64bit RyuJIT; GC = Concurrent Workstation
Mean = 35.8371 us, StdErr = 0.1330 us (0.37%); N = 15, StdDev = 0.5152 us
Min = 35.0340 us, Q1 = 35.3669 us, Median = 35.8602 us, Q3 = 36.3705 us, Max = 36.5768 us
IQR = 1.0036 us, LowerFence = 33.8615 us, UpperFence = 37.8760 us
ConfidenceInterval = [35.2863 us; 36.3878 us] (CI 99.9%), Margin = 0.5508 us (1.54% of Mean)
Skewness = -0.01, Kurtosis = 1.36

Total time: 00:00:49 (49.34 sec)

// * Summary *

BenchmarkDotNet=v0.10.13, OS=ubuntu 16.04
Intel Core i5-2320 CPU 3.00GHz (Sandy Bridge), 1 CPU, 4 logical cores and 4 physical cores
.NET Core SDK=2.1.4
  [Host] : .NET Core 2.0.5 (CoreCLR 4.6.0.0, CoreFX 4.6.26018.01), 64bit RyuJIT
  Core   : .NET Core 2.0.5 (CoreCLR 4.6.0.0, CoreFX 4.6.26018.01), 64bit RyuJIT

Job=Core  Runtime=Core  

                            Method |     Mean |     Error |    StdDev | Rank |
---------------------------------- |---------:|----------:|----------:|-----:|
 TestIntegrationWithResourceBounds | 35.84 us | 0.5508 us | 0.5152 us |    1 |

// * Legends *
  Mean   : Arithmetic mean of all measurements
  Error  : Half of 99.9% confidence interval
  StdDev : Standard deviation of all measurements
  Rank   : Relative position of current benchmark mean among all benchmarks (Arabic style)
  1 us   : 1 Microsecond (0.000001 sec)

// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *
BuildPlots.log 
Error: package or namespace load failed for ‘dplyr’ in library.dynam(lib, package, package.lib):
 shared object ‘dplyr.so’ not found
In addition: Warning message:
S3 methods ‘[.fun_list’, ‘[.grouped_df’, ‘all.equal.tbl_df’, ‘anti_join.data.frame’, ‘anti_join.tbl_df’, ‘arrange.data.frame’, ‘arrange.default’, ‘arrange.grouped_df’, ‘arrange.tbl_df’, ‘arrange_.data.frame’, ‘arrange_.tbl_df’, ‘as.data.frame.grouped_df’, ‘as.data.frame.rowwise_df’, ‘as.data.frame.tbl_cube’, ‘as.data.frame.tbl_df’, ‘as.table.tbl_cube’, ‘as.tbl.data.frame’, ‘as.tbl.tbl’, ‘as.tbl_cube.array’, ‘as.tbl_cube.data.frame’, ‘as.tbl_cube.matrix’, ‘as.tbl_cube.table’, ‘as_data_frame.grouped_df’, ‘as_data_frame.tbl_cube’, ‘auto_copy.tbl_cube’, ‘auto_copy.tbl_df’, ‘cbind.grouped_df’, ‘collapse.data.frame’, ‘collect.data.frame’, ‘common_by.NULL’, ‘common_by.character’, ‘common_by.default’, ‘common_by.list’, ‘compute.data.frame’, ‘copy_to.DBIConnection’, ‘copy_to.src_local’, ‘default_missing.data.frame’, ‘default_missing.default’, � [... truncated] 
Execution halted
lsb_release and uname 
➜  ~ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:    16.04
Codename:   xenial
➜  ~ uname -a
Linux gntpcv2 4.13.0-37-generic #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
➜  ~
/usr/local/lib/R/site-library 
$ pwd
/usr/local/lib/R/site-library/dplyr
$ ls -lha libs
total 31M
drwxrwxr-x  2 root staff 4.0K Mar 18 14:07 .
drwxrwxr-x 10 root staff 4.0K Mar 18 14:07 ..
-rwxrwxr-x  1 root staff  31M Mar 18 14:07 dplyr.so
$ file libs/dplyr.so
libs/dplyr.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=867cd963410424e0180f203f8e3ee45298d2c382, not stripped

Any ideas? I am not familiar with R.

lol768 commented 6 years ago

Fixed with R_LIBS_USER=/usr/local/lib/R/

AndreyAkinshin commented 6 years ago

@lol768, thanks for the report. I added notes about your case in FAQ. Also I'm going to investigate all possible issues with RPlotExporter (e.g., see https://github.com/dotnet/BenchmarkDotNet/issues/641) and provide detailed troubleshooting guide in the documentation (as soon as I find time for it).