Inside: benchmark outliers on MacOS

wukefe commented 6 years ago

Three outliers

crc, fft, and nqueens.

Details

mbp-2013 (David's)

Benchmarks	Firefox 39	Firefox 57	Chrome	GCC
crc	4.6952s	2.9793s	1.0698s	0.95s
fft	8.0984s	1.3760s	6.0469s	1.54s
nqueens	12.5489s	6.4014s	7.0794s	3.26s

mbp-2015 (Hanfeng's)

Benchmarks	Firefox 39	Firefox 57	Chrome	GCC
crc	2.8634s	2.7649s	0.9359s	0.83s
fft	4.8199s	1.2897s	5.8782s	1.30s
nqueens	7.9974s	6.2928s	6.5421s	3.22s

Explanations

In crc, the operator xor (^) is optimized in Chrome, but not Firefox

    crc = crc32Lookup[7 * 256 + (one & 0xFF)] ^
      crc32Lookup[6 * 256 + ((one >>> 8) & 0xFF)] ^
      crc32Lookup[5 * 256 + ((one >>> 16) & 0xFF)] ^
      crc32Lookup[4 * 256 + (one >>> 24)] ^
      crc32Lookup[3 * 256 + (two & 0xFF)] ^
      crc32Lookup[2 * 256 + ((two >>> 8) & 0xFF)] ^
      crc32Lookup[1 * 256 + ((two >>> 16) & 0xFF)] ^
      crc32Lookup[two >>> 24]

In fft, the array indexing assignments, in a recursive function fftSimple, are expensive for Chrome, but not Firefox

    for(var k=0; k < N/2; ++k){
        er[k] = r[2*k];
        ei[k] = i[2*k];
        dr[k] = r[2*k + 1]; 
        di[k] = i[2*k + 1]; 
    }

In nqueens, it has relatively close performance between two versions of Firefox on the mbp-2015 that is quite different from the result of the mbp-2013. We think it may have hardware improvement since the mbp-2013.

ljhendren commented 6 years ago

This is very interesting .... but I am a bit worried about the differences in the performance of firefox 39 on the different mbp's. Are we entirely sure that we are running exactly the same firefox 39?

ljhendren commented 6 years ago

How hard would it be to include the more recent mbp in our geomeans when comparing old vs new? We don't use the old firefox for any of the other graphs, but it is interesting for this one ...

wukefe commented 6 years ago

How hard would it be to include the more recent mbp in our geomeans when comparing old vs new? We don't use the old firefox for any of the other graphs, but it is interesting for this one ...

Rerunning all benchmarks on my computer can be done in a reasonable period of time (maybe a few hours). I will let you and @dherre3 decide whether using the data from the newer mbp in the paper.

ljhendren commented 6 years ago

Are we sure that the software setup - Firefox and any associated libraries - are identical?

wukefe commented 6 years ago

Yes, I think so. David and I worked together last night.

Configurations

Firefox (39.0/mac/en-GB) downloaded from https://ftp.mozilla.org/pub/firefox/releases/
Benchmark configuration files downloaded from our GitHub repositories (e.g. wu-wei and Ostrich)

ljhendren commented 6 years ago

Does firefox link to any other native libraries - which might be more current on your machine than David's?

wukefe commented 6 years ago

mbp-2015

Here is the information of dynamically linked libraries for Firefox 39.

HF:MacOS> otool -L firefox
firefox:
    /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 17.0.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/ExceptionHandling.framework/Versions/A/ExceptionHandling (compatibility version 1.0.0, current version 10.0.0)
    @executable_path/libmozglue.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.0.0)
    /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 53.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 635.0.0)

Here is the information of dynamically linked libraries for Firefox 57.

HF:MacOS> otool -L firefox
firefox:
    /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 22.0.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 492.0.0)
    /System/Library/Frameworks/ExceptionHandling.framework/Versions/A/ExceptionHandling (compatibility version 1.0.0, current version 11.0.0)
    @executable_path/libmozglue.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1258.1.0)

I will check with David to see if his computer (mbp-2013) provides different libraries.

dherre3 commented 6 years ago

Firefox39 MacBook Pro 2013

firefox:
    /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 17.0.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/ExceptionHandling.framework/Versions/A/ExceptionHandling (compatibility version 1.0.0, current version 10.0.0)
    @executable_path/libmozglue.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.0.0)
    /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 53.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 635.0.0)

Firefox 57 MacBook Pro 2013

firefox:
    /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 22.0.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 492.0.0)
    /System/Library/Frameworks/ExceptionHandling.framework/Versions/A/ExceptionHandling (compatibility version 1.0.0, current version 11.0.0)
    @executable_path/libmozglue.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1258.1.0)

GCC:

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/c++/4.2.1
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Sable / dls18-ostrich