`pkg-config` does not update MacOS SDK paths after major macOS upgrades

shenlebantongying commented 1 year ago

`brew doctor` output

It is single warning that is impossible to deal as of right now.

> brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  openssl@1.1

Verification

[X] My "brew doctor output" above says Your system is ready to brew. and am still able to reproduce my issue.
[X] I ran brew update twice and am still able to reproduce my issue.
[X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

`brew config` output

> brew config
HOMEBREW_VERSION: 4.1.17
ORIGIN: https://github.com/Homebrew/brew
HEAD: 35746e0a6ba6c3c5cfe56d99f79d9ec9f52ee15f
Last commit: 3 days ago
Core tap JSON: 25 Oct 01:23 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.RB7odOrKDO/org.xquartz:0
HOMEBREW_EDITOR: nano
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit arm_firestorm_icestorm
Clang: 15.0.0 build 1500
Git: 2.39.3 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 8.1.2 => /usr/bin/curl
macOS: 14.0-arm64
CLT: 15.0.0.0.1.1694021235
Xcode: 15.0.1
Rosetta 2: false

What were you trying to do (and why)?

Trying to use pkg-config via cmake to find pango which depends on libffi from macOS SDK, but failed because pkg-config will provides invalid path pointing to older macOS SDK

Minimum CMakeLists.txt to reproduce the problem:

cmake_minimum_required(VERSION 3.27)
project(test)

set(CMAKE_C_STANDARD 23)
find_package(PkgConfig REQUIRED)
pkg_check_modules(pango IMPORTED_TARGET pango)
add_executable(main main.c)
target_link_libraries(main PkgConfig::pango)

Will gives

CMake Error in CMakeLists.txt:
  Imported target "PkgConfig::pango" includes non-existent path

    "/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/include/ffi"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.

What happened (include all command output)?

The pkg-config tries to uses libffi.pc from macOS 11 while i am using the current Xcode and macOS 14.

The pkg-config will give cmake a none-exists path because this line got included:

https://github.com/Homebrew/brew/blob/master/Library/Homebrew/os/mac/pkgconfig/11/libffi.pc#L1

instead of

https://github.com/Homebrew/brew/blob/master/Library/Homebrew/os/mac/pkgconfig/14/libffi.pc#L1

The pkg-config's path falsely includes /opt/homebrew/Library/Homebrew/os/mac/pkgconfig/11

pkg-config --variable pc_path pkg-config
/opt/homebrew/lib/pkgconfig:/opt/homebrew/share/pkgconfig:/usr/local/lib/pkgconfig:/usr/lib/pkgconfig:/opt/homebrew/Library/Homebrew/os/mac/pkgconfig/11

What did you expect to happen?

I think pkg-config's search path should be /opt/homebrew/Library/Homebrew/os/mac/pkgconfig/14 instead of /opt/homebrew/Library/Homebrew/os/mac/pkgconfig/11 on newest macOS & XCode.

Step-by-step reproduction instructions (by running `brew` commands)

brew install pkg-config
pkg-config --variable pc_path pkg-config

The path outputted should includes /opt/homebrew/Library/Homebrew/os/mac/pkgconfig/14 instead of /opt/homebrew/Library/Homebrew/os/mac/pkgconfig/11

For extra info, my XCode 15 installtion includes those things, the 11 got removed recently? i don't know.

/Library/Developer/CommandLineTools/SDKs
> ls -al
total 0
drwxr-xr-x  9 root  wheel  288 27 Sep 10:50 ./
drwxr-xr-x  5 root  wheel  160 27 Sep  2022 ../
lrwxr-xr-x  1 root  wheel   14 27 Sep 10:50 MacOSX.sdk@ -> MacOSX14.0.sdk
drwxr-xr-x  7 root  wheel  224 24 Jun 11:04 MacOSX12.3.sdk/
lrwxr-xr-x  1 root  wheel   14 24 Jun 11:03 MacOSX12.sdk@ -> MacOSX12.3.sdk
drwxr-xr-x  7 root  wheel  224 27 Sep 10:50 MacOSX13.3.sdk/
lrwxr-xr-x  1 root  wheel   14 27 Sep 10:49 MacOSX13.sdk@ -> MacOSX13.3.sdk
drwxr-xr-x  7 root  wheel  224 31 Aug 15:12 MacOSX14.0.sdk/
lrwxr-xr-x  1 root  wheel   14 27 Sep 10:49 MacOSX14.sdk@ -> MacOSX14.0.sdk

Bo98 commented 1 year ago

Were you previously running macOS 11? Currently you need to brew reinstall pkg-config when you do major macOS updates.

shenlebantongying commented 1 year ago

@Bo98 Yes, it fixed the problem, thank you very much :)

Were you previously running macOS 11?

I forget. I got my machine around 2021 Sept, it might be, but I upgraded every versions in between.

However this is so frustrating. The process of deep diving into pkg-config to find a workaround is so long :(

Should this be closed? Or i just change the title.

carlocab commented 1 year ago

I tried to fix the handling of this a while back, but ran out of steam. I could try looking at it again, though I'm happy for someone else to take a stab at it.

We can keep this issue open for now.

alebcay commented 1 year ago

I'm interested in having a stab at this, but I'm not sure what the best path would be to fix it. The search path for pkg-config is baked into pkg-config binary at build time, currently set to include #{HOMEBREW_LIBRARY}/Homebrew/os/mac/pkgconfig/#{MacOS.version}.

I would propose that we instead include something like #{HOMEBREW_LIBRARY}/Homebrew/os/mac/pkgconfig/current, which will be a symlink to #{MacOS.version}. (The symlink should not be checked into Git - or we can put this symlink somewhere else - maybe somewhere in Cellar? - to avoid this.)

Then the question would be at what point to update the symlink to point at the current version of #{HOMEBREW_LIBRARY}/Homebrew/os/mac/pkgconfig/#{MacOS.version}. Maybe:

Provide a command for this. brew reconfigure-pkg-config or something.
- If we see this pattern with more packages, perhaps its worth making this a more general solution, something like dpkg-reconfigure - brew reconfigure pkg-config. This would probably need some DSL additions for more formulae to be able to specify/leverage this.
- From UX perspective, I think this is not much of an improvement over brew reinstall pkg-config. User still needs to discover the command, and run it themselves.
Build this check into some brew command that runs often. Somewhere in brew update maybe?
- Overloading update to do other tasks might be a bad pattern/smell, especially since OS upgrades don't happen often.
- User could still be stuck with broken/impaired pkg-config until the brew command runs.
Add a launchd item to contain this and possible future things that need to run outside of the lifecycle of brew commands (e.g. once per boot or something).
- Need something Linux-equivalent. systemd probably covers most people, but maybe provide a manual option/command that users can hook into their own preferred init systems.
- Expands the scope of when/how brew-related code is running on the system. Also one more thing to clean up on uninstall, and not contained neatly in HOMEBREW_PREFIX.
Something else...?

Or maybe there's a better way than this symlink approach, but for most other ideas I can think of, I still see the same issue of user needing to manually run a command (maybe brew can prompt/show a message, but still requires user to run a brew command) or we need some way to automate/do stuff at the right time.

Bo98 commented 1 year ago

Alternative, which could apply to non-pkg-config cases like GCC and Clang's DEFAULT_SYSROOT, could be a way of mark formulae that need reinstalling on newer macOS and those will appear as "outdated" if current OS != tab.dig("built_on", "os_version")

carlocab commented 1 year ago

I would propose that we instead include something like #{HOMEBREW_LIBRARY}/Homebrew/os/mac/pkgconfig/current, which will be a symlink to #{MacOS.version}. (The symlink should not be checked into Git - or we can put this symlink somewhere else - maybe somewhere in Cellar? - to avoid this.)

This was essentially the approach I took in https://github.com/Homebrew/brew/pull/13664. Fine with doing this, but I remember there being concerns about when that symlink would be updated.

MikeMcQuaid commented 1 year ago

Alternative, which could apply to non-pkg-config cases like GCC and Clang's DEFAULT_SYSROOT, could be a way of mark formulae that need reinstalling on newer macOS and those will appear as "outdated" if current OS != tab.dig("built_on", "os_version")

This is the approach I'd like to see: a new DSL that forces an upgrade or reinstall in the case mentioned here.

MikeMcQuaid commented 2 months ago

@Bo98 @carlocab do we want to still do anything here? Given we've done another major release without this: it seems we don't really want/need to do anything here.

cho-m commented 2 months ago

I do think the DSL to force reinstall would be useful.

A temporary alternative would be a message in brew doctor for formulae that we are aware are incompatible and need reinstall. Should be easier to implement and at least provide users some information on what is problem.

Bo98 commented 2 months ago

Given we've done another major release without this: it seems we don't really want/need to do anything here.

We've kinda gotten lucky given the last couple macOS releases have been very minor in terms of dependency updates.

But it is a problem if you go two major Xcode versions without reinstalling.

There is some potential for seeing if pkgconf can handle this better but I'm still scoping that out.

MikeMcQuaid commented 2 months ago

@cho-m Either of those seem good. Would you be able to pick something up?

Homebrew / brew