cmake: Disable discovery of Homebrew libraries for dependencies

PatTheMav commented 1 week ago

Description

Adds the default installation location of Homebrew packages to list of ignored paths for CMake to resolve dependencies on macOS.

Motivation and Context

macOS builds should only use dependencies built via the obs-deps build scripts. Default variants of the same dependencies are not compatible with our app packaging requirements and thus will create issues when creating the app bundle.

This is specifically an issue when MbedTLS is installed via Homebrew which ships a CMake package config by default and is picked up by our code ever since we switched to prefer CMake packages.

How Has This Been Tested?

Tested on macOS 15 with MbedTLS installed via Homebrew and observed that CMake will only pick up MbedTLS provided by obs-deps.

Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

[x] My code has been run through clang-format.
[x] I have read the contributing document.
[x] My code is not on the master branch.
[x] The code has been tested.
[x] All commit messages are properly formatted and commits squashed where appropriate.
[x] I have included updates to all appropriate documentation.

jcm93 commented 1 week ago

This is still pulling in some homebrew libraries, for example libluajit is still pulled from /opt/homebrew/Cellar if it exists there. The --debug-find output reveals this:

CMake Debug Log at cmake/finders/FindLuajit.cmake:95 (find_library):
  find_library called with the following settings:

    VAR: Luajit_LIBRARY
    NAMES: "luajit"
           "luajit-51"
           "luajit-5.1"
           "lua51"
    Documentation: Luajit location
    Framework
      Only Search Frameworks: 0
      Search Frameworks Last: 0
      Search Frameworks First: 1
    AppBundle
      Only Search AppBundle: 0
      Search AppBundle Last: 0
      Search AppBundle First: 1
    CMAKE_FIND_USE_CMAKE_PATH: 1
    CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH: 1
    CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH: 1
    CMAKE_FIND_USE_CMAKE_SYSTEM_PATH: 1
    CMAKE_FIND_USE_INSTALL_PREFIX: 1

  find_library considered the following locations:

    /Users/xxx/code/obs-studio/.deps/obs-deps-2024-09-05-universal/lib/libluajit(\.tbd|\.dylib|\.so|\.a)
    /Users/xxx/code/obs-studio/.deps/obs-deps-2024-09-05-universal/libluajit(\.tbd|\.dylib|\.so|\.a)
    /Users/xxx/code/obs-studio/.deps/obs-deps-qt6-2024-09-05-universal/lib/libluajit(\.tbd|\.dylib|\.so|\.a)
    /Users/xxx/code/obs-studio/.deps/obs-deps-qt6-2024-09-05-universal/libluajit(\.tbd|\.dylib|\.so|\.a)

  The item was found at

    /opt/homebrew/Cellar/luajit/2.1.1725453128/lib/libluajit.dylib

I'll see if I can dig into it any more... perhaps /opt/homebrew is still in the SYSTEM_ENVIRONMENT_PATH or CMAKE_SYSTEM_PATH and so it's still searched?

PatTheMav commented 1 week ago

This is still pulling in some homebrew libraries, for example libluajit is still pulled from /opt/homebrew/Cellar if it exists there. The --debug-find output reveals this:

CMake Debug Log at cmake/finders/FindLuajit.cmake:95 (find_library):
  find_library called with the following settings:

    VAR: Luajit_LIBRARY
    NAMES: "luajit"
           "luajit-51"
           "luajit-5.1"
           "lua51"
    Documentation: Luajit location
    Framework
      Only Search Frameworks: 0
      Search Frameworks Last: 0
      Search Frameworks First: 1
    AppBundle
      Only Search AppBundle: 0
      Search AppBundle Last: 0
      Search AppBundle First: 1
    CMAKE_FIND_USE_CMAKE_PATH: 1
    CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH: 1
    CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH: 1
    CMAKE_FIND_USE_CMAKE_SYSTEM_PATH: 1
    CMAKE_FIND_USE_INSTALL_PREFIX: 1

  find_library considered the following locations:

    /Users/xxx/code/obs-studio/.deps/obs-deps-2024-09-05-universal/lib/libluajit(\.tbd|\.dylib|\.so|\.a)
    /Users/xxx/code/obs-studio/.deps/obs-deps-2024-09-05-universal/libluajit(\.tbd|\.dylib|\.so|\.a)
    /Users/xxx/code/obs-studio/.deps/obs-deps-qt6-2024-09-05-universal/lib/libluajit(\.tbd|\.dylib|\.so|\.a)
    /Users/xxx/code/obs-studio/.deps/obs-deps-qt6-2024-09-05-universal/libluajit(\.tbd|\.dylib|\.so|\.a)

  The item was found at

    /opt/homebrew/Cellar/luajit/2.1.1725453128/lib/libluajit.dylib

I'll see if I can dig into it any more... perhaps /opt/homebrew is still in the SYSTEM_ENVIRONMENT_PATH or CMAKE_SYSTEM_PATH and so it's still searched?

Can you try with it changed to CMAKE_IGNORE_PATH?

jcm93 commented 1 week ago

No change with CMAKE_IGNORE_PATH instead of CMAKE_IGNORE_PREFIX_PATH.

jcm93 commented 1 week ago

From https://cmake.org/cmake/help/latest/variable/CMAKE_IGNORE_PATH.html:

The listed directories do not apply recursively, so any subdirectories to be ignored must also be explicitly listed.

So maybe CMAKE_IGNORE_PATH will work if we explicitly enumerate /opt/homebrew/Cellar/lib, /opt/homebrew/lib, etc.

PatTheMav commented 1 week ago

From https://cmake.org/cmake/help/latest/variable/CMAKE_IGNORE_PATH.html:

The listed directories do not apply recursively, so any subdirectories to be ignored must also be explicitly listed.

So maybe CMAKE_IGNORE_PATH will work if we explicitly enumerate /opt/homebrew/Cellar/lib, /opt/homebrew/lib, etc.

That does make it sound like CMAKE_IGNORE_PREFIX_PATH is more correct:

CMAKE_IGNORE_PREFIX_PATH provides a more appropriate way to ignore a whole search prefix.

jcm93 commented 1 week ago

Hmm, for the prefix path, it also says:

The prefixes are also ignored by the Config mode of the find_package() command (Module mode is unaffected)

Aren't OBS's CMake finders in the module mode? So find_package will ignore this?

PatTheMav commented 1 week ago

Hmm, for the prefix path, it also says:

The prefixes are also ignored by the Config mode of the find_package() command (Module mode is unaffected)

Aren't OBS's CMake finders in the module mode? So find_package will ignore this?

To me it sounds like the find_package call in find modules is unaffected by this, but we commonly only use that one to find pkg-config - the dependencies themselves are resolved via find_library and find_path and their docs state that the variable is used.

jcm93 commented 1 week ago

Per a CMake maintainer, /opt/homebrew is considered a separate installation prefix to /opt/homebrew/Cellar. So I think we also need to separately add /opt/homebrew/Cellar to the ignore prefix path list, and perhaps instead of /opt/homebrew, /opt/homebrew/lib?

jcm93 commented 1 week ago

It seems like if we exclude all of the following:

list(APPEND CMAKE_IGNORE_PREFIX_PATH "/opt/homebrew/lib" "/usr/local/lib" "/opt/homebrew" "/usr/local")

Then CMake successfully stops finding any homebrew versions of MbedTLS's config on my system.

The extra exclusions seem to be necessary because cmFindPackageCommand::SearchPrefix is appending "/lib/cmake/MbedTLS" to all prefixes for MbedTLS after checking against ignored prefixes, so if /usr/local/ is anywhere in our system environment paths, it can end up searching /usr/local/lib/cmake/MbedTLS, and finding it, even though /usr/local/lib is specifically excluded.

jcm93 commented 1 week ago

Looking at it one more time, my suggested /opt/homebrew/lib and /usr/local/lib exclusions may actually be entirely superfluous, since those are not actually real "prefixes" in the CMake/build system sense of the term (i.e. a CMake prefix derived by looking at $PATH and removing the last component, as is done to yield /opt/homebrew, I do not think should ever yield /opt/homebrew/lib, under normal circumstances).

So it ought to be sufficient to just exclude /opt/homebrew as well as /usr/local. I can at least confirm that on my system, with only those two exclusions, we don't find any homebrew libraries, including the problematic arm64 or x86_64 versions of either LuaJIT or MbedTLS.

obsproject / obs-studio