Investigate performance differences between Xcode `clang` and `homebrew` `clang`

[wbkboyer]

Background

While writing the environment setup instructions for MacOS on 2. Dev Setup, I discovered that some of the instructions I'd included pertained to using the clang installed via homebrew, i.e. adding the following to one's ~/.zshrc file:

    export PATH="/opt/homebrew/opt/llvm/bin:$PATH"

    export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
    export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"

Before removing these instructions, I wanted to make sure there that instructing users to only rely on the Xcode clang was the correct course of action in terms of performance of the resulting executable (see this discussion, where they say "For an M1 machine, an Apple Clang build has 70% lower performance than an LLVM 16 build" for their particular application).

Problem statement

Determine whether there is sufficient motivation to instruct users to install clang via homebrew rather than to use the clang bundled with the Xcode command-line tools.

[wbkboyer]

I compiled two different versions of the planarity executable using the homebrew clang vs. Xcode clang:

/opt/homebrew/opt/llvm/bin/clang -O3 ../c/planarityApp/**.c ../c/graphLib/**.c ../c/graphLib/extensionSystem/**.c ../c/graphLib/homeomorphSearch/**.c ../c/graphLib/io/**.c ../c/graphLib/lowLevelUtils/**.c ../c/graphLib/planarityRelated/**.c -o planarity-brew

/usr/bin/clang -O3 ../c/planarityApp/**.c ../c/graphLib/**.c ../c/graphLib/extensionSystem/**.c ../c/graphLib/homeomorphSearch/**.c ../c/graphLib/io/**.c ../c/graphLib/lowLevelUtils/**.c ../c/graphLib/planarityRelated/**.c -o planarity-xcode

Then I ran on all 12,005,168 graphs with N=10:

wbkboyer@Wandas-MacBook-Pro Release % ./planarity-brew -t -p n10_all.g6 n10_all-brew.p.out.txt
Start testing all graphs in "n10_all.g6".

Done testing all graphs (20.577 seconds).

wbkboyer@Wandas-MacBook-Pro Release % ./planarity-xcode -t -p n10_all.g6 n10_all-xcode.p.out.txt
Start testing all graphs in "n10_all.g6".

Done testing all graphs (21.005 seconds).

This raw execution speed difference is not appreciable, and so I generated all 37,433,760 graphs for N=11 and M=22

% ../../../git/nauty/geng 11 22:22 > n11-m22.g6
>A ../../../git/nauty/geng -d0D10 n=11 e=22
>Z 37433760 graphs generated in 9.58 sec

And executed again:

wbkboyer@Wandas-MacBook-Pro Release % ./planarity-brew -t -p n11-m22.g6 n11-m22-brew.p.out.txt
Start testing all graphs in "n11-m22.g6".

Done testing all graphs (101.211 seconds).

wbkboyer@Wandas-MacBook-Pro Release % ./planarity-xcode -t -p n11-m22.g6 n11-m22-xcode.p.out.txt
Start testing all graphs in "n11-m22.g6".

Done testing all graphs (104.981 seconds).

There does seem to be a slight slowdown (~3%) for the executable compiled using Xcode clang vs. homebrew clang, but I'm not sure if this is sufficient motivation to provide alternate install instructions.

Other motivators would include dependencies not being included in Xcode clang (e.g. this homebrew issue requests the use of LLVM 15 instead of Xcode to enable use of OpenMP for multithreading), but as far as I can tell we don't have that issue...

[john-boyer-phd]

Excellent work determining the timing difference. Now that we see the result, it is negligible in my view, so I would rather we have the simpler instructions for XCode and people can do their own work to use the homebrew install instead, if they want.