pkg-config files on >=10.14 assume headers in /usr/include

[railwaycat]

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

• [x] are reporting a bug others will be able to reproduce and not asking a question. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh
• [x] ran a brew command and reproduced the problem with multiple formulae? If it's a problem with a single, official formula (not cask) please file this issue at Homebrew/homebrew-core: https://github.com/Homebrew/homebrew-core/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
• [x] ran brew update and can still reproduce the problem?
• [x] ran brew doctor, fixed all issues and can still reproduce the problem?
• [x] ran brew config and brew doctor and included their output with your issue?

  
  % brew doctor
  Please note that these warnings are just used to help the Homebrew maintainers
  with debugging if you file an issue. If everything you use Homebrew for is
  working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: The following directories do not exist: /usr/local/sbin

You should create these directories and change their ownership to your account. sudo mkdir -p /usr/local/sbin sudo chown -R $(whoami) /usr/local/sbin % brew config HOMEBREW_VERSION: 1.7.6-143-gc6acab6 ORIGIN: https://github.com/Homebrew/brew HEAD: c6acab66feb63743fe69b38dab7dc7b90710e838 Last commit: 2 hours ago Core tap ORIGIN: https://github.com/Homebrew/homebrew-core Core tap HEAD: 537f3703aa41c550edbaaf28dd280c6ec929f946 Core tap last commit: 34 minutes ago HOMEBREW_PREFIX: /usr/local HOMEBREW_DEV_CMD_RUN: 1 CPU: octa-core 64-bit kabylake Homebrew Ruby: 2.3.7 => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby Clang: 10.0 build 1000 Git: 2.17.1 => /Applications/Xcode.app/Contents/Developer/usr/bin/git Curl: 7.54.0 => /usr/bin/curl macOS: 10.14-x86_64 CLT: 10.0.0.0.1.1535735448 Xcode: 10.0 XQuartz: 2.7.11 => /opt/X11

To help us debug your issue please explain:
- What you were trying to do (and why)
I tried to build a package which uses `pkg-config` from Homebrew.

- What happened (include command output)
I met an error when build:

fatal error: 'libxml/tree.h' file not found

include <libxml/tree.h>

     ^~~~~~~~~~~~~~~

1 error generated.

It looks like `pkg-config` from Homebrew is not providing the correct include directory of libs shipping with macOS(Xcode and CLI tools). As the release note of Xcode 10, header files which were under `/usr/include` no longer exists and been moved to Xcode: `/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include` and CLI tools: `/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include`.

output:

% which pkg-config /usr/local/bin/pkg-config

% pkg-config --cflags --libs "libxml-2.0 > 2.6.17" -I/usr/include/libxml2 -lxml2

% pkg-config --variable pcfiledir "libxml-2.0 > 2.6.17" /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.14

% cat /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.14/libxml-2.0.pc prefix=/usr exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include modules=1

Name: libXML Version: 2.9.4 Description: libXML library version2. Requires: Libs: -L${libdir} -lxml2 Libs.private: -lz -lpthread -licucore -lm Cflags: -I${includedir}/libxml2

I tested only with `libxml2` but I believe this issue affect all libs which have `pkg-config` file from Homebrew `Library/Homebrew/os/mac/pkgconfig/10.14`, they are: libcurl.pc, libexslt.pc, libxml-2.0.pc, libxslt.pc, sqlite3.pc, zlib.pc.
- What you expected to happen

% pkg-config --cflags --libs "libxml-2.0 > 2.6.17"

should gives the correct path to the header files under Xcode or CLI tools. 

- Step-by-step reproduction instructions (by running `brew` commands)
1. `brew install pkg-config`
2. `pkg-config --cflags --libs "libxml-2.0 > 2.6.17"`

[MikeMcQuaid]

We supply a default homebrew_sdkroot to the *.pc files. This would be the CLT so Xcode-only systems would not be supported.

• This however can be overridden by the superenv shim so Xcode-only support can be retained there.

@Bo98 I like the sound of that. Having it be CLT-only seems fine (particularly given this was already broken on Xcode-only and no-one complained).

We create a system where pkg-config is marked as outdated if it was built on a different major OS version than it was installed. I'm not talking about HOMEBREW_SKIP_OR_LATER_BOTTLES (though that would be a part of it), but rather if at any point a user upgrades macOS (excluding minor updates) then brew will immediately mark pkg-config as outdated. This would require to somewhere store the version of macOS used to build pkg-config - which may have been installed from source or from a bottle.

I think best we can probably do here is something in test do which necessitates a revision and a pour_bottle block which means the bottle isn't poured on newer versions. Beyond this we could consider an additional bottle DSL that indicates that this bottle won't work on later macOS versions.

• Then we can bake the Homebrew *.pc path into pkg-config as we were before (so basically partially reverting Homebrew/homebrew-core#52565).

Seems fine given the above 👍

[MikeMcQuaid]

That said: if we get large numbers of complaints/issues: we can consider another option.

To be explicit: I would consider us to now have "a large (enough) number of complaints".

[Bo98]

Just to be clear:

which necessitates a revision and a pour_bottle block

A pour_bottle will help new installations. A revision will be limited help as the problem isn’t the bottles themselves but prompting existing installations to reinstall at the right time - this isn’t something detectable by the CI since the concern is existing installations more than new installations. If we start adding fixed SDK locations, users would be required to brew reinstall pkg-config when upgrading macOS to prevent hard-to-debug errors. Most people will not do this.

[MikeMcQuaid]

A revision will be limited help as the problem isn’t the bottles themselves but prompting existing installations to reinstall at the right time - this isn’t something detectable by the CI since the concern is existing installations more than new installations.

You're right. The best bet here is just a brew doctor special-case, I think, which can also be run as a pre-install check (or just a pre-install check if anything in the dependency tree uses pkg-config).

[Bo98]

as a pre-install check

At that point you are one step away from making it a brew upgrade/outdated check and automating the reinstall.

special-case

Yeah, such check would probably be something like checking the output of pkg-config --variable=homebrew_sdkroot libxml-2.0, unless there's some other information somewhere that can be read which I'm not aware of.

[Bo98]

That output check actually will not work well if anyone had brew link libxml2. Argh, worst case scenario we touch some file in the install directory and check that.

[MikeMcQuaid]

At that point you are one step away from making it brew upgrade/outdated check and automating the reinstall.

Yep, I'm not opposed to that. Given that: it could be something we (ab)use the linkage checker checks to do.

Argh, worst case scenario we touch some file in the install directory and check that.

Yup. I wouldn't worry too much about the OS upgrade case yet, though, it's not as pressing (and can be pulled into a new issue).

[Bo98]

I wouldn't worry too much about the OS upgrade case yet, though, it's not as pressing

Yeah, the rest of what needs done is fairly simple.

One more thought on it though: any thoughts on adding the OS pkg-config was built on to the install receipt and using that?

[MikeMcQuaid]

One more thought on it though: any thoughts on adding the OS pkg-config was built on to the install receipt and using that?

That seems like a good idea (for all install receipts). Could also stick something like Xcode/CLT/SDK versions if you anticipate it being useful for fixing bugs in future.

[whitty]

I don't want to wade in in the wrong place, but my ticket #7292 was closed so I can't ask there.

I know this is an evolving situation, but this ticket is full of detailed discussion that is hard to parse for someone not knee deep in it. Could you perhaps synthesise a post with

• current state
• known workarounds/reverts
• expectation of duration (ie so people can decide whether to wait for a fix or try to revert)

and post it to each of the tickets you've closed, so that Google gives good results, but you don't get flooded with dumb questions on this thread.

PS:

(reverts especially welcome, because the core update seemed to break compatibility with existing bottles (eg python2), and since its discontinued (but that was hidden) I have also to fix up a whole legacy build-system in addition to pkg-config woes)

[MikeMcQuaid]

@whitty We have reverted the changes to pkg-config (brew upgrade) until we are ready with the new approach (which will continue to use Homebrew pkg-config files).

[0xTim]

Hi all, Posting here as it seems the most relevant, but feel free to tell me to open a new issue! TL;DR - #7277 has broken every Swift Package Manager project that relies on a system library.

So in short, that change seems to be adding the homebrew_sdkroot path to the includes directory, both in Xcode and when building from the command line when using SwiftPM. This seems to affect projects in Xcode if you open the manifest directory or use swift package generate-xcodeproj. You can see below the changes added to Xcode

This ends up causing long build errors all based around redefinition of modules as they're included by both Xcode and the command line tools. Is there a short workaround for this? I'm assuming this is a Homebrew issue as the above changed fixed it.

Thanks!

(PS there are also a couple of forum posts here and here tracking the issue.

[MikeMcQuaid]

TL;DR - #7277 has broken every Swift Package Manager project that relies on a system library.

The approach has changed multiple times since then. Additionally, Xcode and Swift don't provide or use pkg-config by default and certainly don't use Homebrew's packages. This is likely instead an issue with your specific project(s).

If you can reproduce an issue with actually running Homebrew commands or provided packages: we can investigate.

[dlbuckley]

This is likely instead an issue with your specific project(s).

Unfortunately this isn't the case, all of the projects were running fine until this (or related) change came in. One day it worked, and then after updating homebrew the exact same project with no changes, failed. This was then verified across multiple swift package projects that import system libraries.

Can I suggest this change is reverted until we get to the root of exactly what's going on as its essentially halted production of our product as there doesn't seem to be a way to downgrade homebrew to an older version to continue work. Unless you have any other suggestions @MikeMcQuaid ?

[0xTim]

@MikeMcQuaid SwiftPM relies on pkg-config to tell it what includes paths to use and what linker paths to use, as described here. So it looks whatever changes ended up being final in 2.2.12 are injecting in the CLT in the pkg-config's search path. Trying to work out how that's happening now

[MikeMcQuaid]

Firstly, drive-by 👎s of my comments without addressing my questions do nothing other than to demotivate me to help.

If you can reproduce an issue with actually running Homebrew commands or provided packages: we can investigate.

Can I suggest this change is reverted until we get to the root of exactly what's going on as its essentially halted production of our product as there doesn't seem to be a way to downgrade homebrew to an older version to continue work

If this has halted production of your product: your business needs to figure out how to better deploy your upgrades of open source projects run by volunteers.

Unless you have any other suggestions @MikeMcQuaid ?

Someone needs to provide: step-by-step reproduction instructions (with as minimal input data as possible) running brew and pkg-config commands (not SwiftPM or Xcode).

If we get that, someone can clearly demonstrate the problem and it cannot be fixed without a revert: we'll consider one.

So it looks whatever changes ended up being final in 2.2.12 are injecting in the CLT in the pkg-config's search path.

The CLT does not provide pkg-config, Homebrew provides pkg-config. If SwiftPM is relying on Homebrew's pkg-config to provide output in a format that's not documented or stated anywhere on our end: that's SwiftPM's bug to be resolved.

Come on, folks, we're all (presumably?) engineers here. Put yourselves in our shoes: someone shows up saying "X broke something and needs to be reverted" but provides no reproduction instructions outside of "my project isn't working". Be the bug report you want to see in the world!

[soonho-tri]

FYI, packages using waf + pkg-config are broken.

One example: brew install sratom --build-from-source. It generates the following error message:

Checking for program 'pkg-config'             : /usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config
Checking for 'lv2 >= 1.16.0'                  : not found
The configuration failed

My understanding is that when this line, https://github.com/Homebrew/brew/blob/master/Library/Homebrew/shims/mac/super/pkg-config#L3, is executed, HOMEBREW_OPT is not properly set (because it's called from waf?). As a result, it fails to call the proper pkg-config (which is usually at /usr/local/opt/pkg-config).

There are more examples. Please let me know if you want me to open a separate issue.

/cc @bo98 who created the file, Library/Homebrew/shims/mac/super/pkg-config.

[Bo98]

That's likely a completely separate issue from SwiftPM.

Calling Library/Homebrew/shims/mac/super/pkg-config when you run brew install is intentional. HOMEBREW_OPT is set in the superenv here: https://github.com/Homebrew/brew/blob/e12da9491d2dc557636458c265196fedde34a30a/Library/Homebrew/extend/ENV/super.rb#L54

Is there more verbose logs than "the configuration failed"? That really says nothing about why it failed.

[dlbuckley]

If this has halted production of your product: your business needs to figure out how to better deploy your upgrades of open source projects run by volunteers.

@MikeMcQuaid I don't think this comment was really called for 😞 We are just trying to figure out what's going on and have narrowed it down to some change in homebrew. Comments like this push away people who are actively trying to find issues and contribute.

As for 'our business'; we haven't even reached deployment stage, this is an issue that has effected all of our local development environments.

Regarding reproduction steps; we are trying to figure out a small and simple way to reproduce it on the Vapor discord channel now. You are welcome to join if you like.

[Bo98]

Re: SwiftPM, does this work for you guys? Homebrew/homebrew-core#52820

[soonho-tri]

Is there more verbose logs than "the configuration failed"? That really says nothing about why it failed.

@Bo98, This is part of /private/tmp/sratom-20200410-9155-ehu8vz/sratom-0.6.4/build/config.log.

['/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config', 'lv2 >= 1.16.0', '--cflags', '--libs', 'lv2', '>=', '1.16.0']
not found

Still not useful? So I've modified Library/Homebrew/shims/mac/super/pkg-config to check why it generates not found. As I mentioned above, when this file is executed from waf, the HOMEBREW_OPT is not set and /usr/local/opt/pkg-config is not called. It's true that the environment variable is set correctly when waf is called from homebrew though.

[Bo98]

That's useful. I'll have a look.

[0xTim]

@Bo98 honestly I have no idea! 😅 I'm currently trying to work out how SwiftPM uses pkg-config and how pkg-config for something like /usr/local/Cellar/libressl/3.0.2/lib/pkgconfig/openssl.pc is getting it to include multiple module maps. The errors are lots of variations on:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/module.modulemap:1:8: error: redefinition of module 'AppleTextureEncoder'
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/module.modulemap:1:8: error: redefinition of module 'AppleTextureEncoder'
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/module.modulemap:1:8: error: redefinition of module 'AppleTextureEncoder'
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/module.modulemap:1:8: error: redefinition of module 'AppleTextureEncoder'
module AppleTextureEncoder [system] [extern_c] {
       ^
module AppleTextureEncoder [system] [extern_c] {
       ^
module AppleTextureEncoder [system] [extern_c] {
       ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/module.modulemap:1:8: note: previously defined here
module AppleTextureEncoder [system] [extern_c] {
       ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/module.modulemap:7:8: error: redefinition of module 'Compression'
module Compression [system] [extern_c] {
       ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/module.modulemap:7:8: note: previously defined here
module Compression [system] [extern_c] {
       ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/module.modulemap:13:8: error: redefinition of module 'Darwin'
module AppleTextureEncoder [system] [extern_c] {
       ^

@MikeMcQuaid apologies if I came off as accusationary in my initial comment, to be clear I'm not saying it's definitely a Homebrew error - it could well be SwiftPM using something they shouldn't be, but the change for the multiple people reporting the errors was a Homebrew upgrade and I managed to track down the inclusion of the CLTs, which matches the errors, so Homebrew was my first point to raise it

[Bo98]

honestly I have no idea!

Well I guess we'll find out when I merge the pull request.

getting it to include multiple module maps

Apple's SDK system is extremely fragile, particularly after they moved to shipping /usr/include headers in the SDK.

The way Apple originally designed SDKs and the way system headers are meant to be used are incompatible in many ways - you simply cannot use the correct system headers and the latest SDK at the same time if you are not on the latest OS. Things also break when both the CLT & Xcode are used, despite the SDK contents being identical.

With that said, hopefully the linked pull request fixes the SwiftPM problems. It should have the effect that pkg-config --cflags libcurl returns nothing, but pkg-config --cflags libxml-2.0 does (since libxml isn't in a "standard" location). This should avoid the -I flag appearing in your Xcode builds and make it work out-of-the-box, at the cost that it doesn't enforce the correct SDK should you end up using system libraries like Curl (though it was like that before).

[0xTim]

@Bo98 so after more digging - swift build will run pkg-config --variable pc_path pkg-config before actually building any source files. On 2.2.11, in zlib.pc in /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.15/zlib.pc it contains:

prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
sharedlibdir=${libdir}
includedir=${prefix}/include

Name: zlib
Description: zlib compression library
Version: 1.2.11

Requires:
Libs: -L${libdir} -L${sharedlibdir} -lz
Cflags: -I${includedir}

In 2.2.12, with the changes, the file now looks like

homebrew_sdkroot=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
prefix=${homebrew_sdkroot}/usr
exec_prefix=/usr
libdir=${exec_prefix}/lib
sharedlibdir=${libdir}
includedir=${prefix}/include

Name: zlib
Description: zlib compression library
Version: 1.2.11

Requires:
Libs: -L${libdir} -L${sharedlibdir} -lz
Cflags: -I${includedir}

That would explain why the C flags produced would now include the CLTs

[soonho-tri]

@Bo98 , it seems like my problem is my own as I can't reproduce it on another mac. Sorry for wasting your time.

[Bo98]

@soonho-tri No worries. If you figure out what happened, let me know.

[0xTim]

Additionally, it looks like SwiftPM parses the .pc files to get things like the C flags (see here). I have no experience with pkg-config so no idea if this is correct or not, but pkg-config zlib -cflags does result in -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include

[Bo98]

Yes, and when my pull request is merged, it will then return nothing:

$ pkg-config --cflags zlib

It will basically filter out <SDK>/usr/include.

Out of the ones Homebrew ship, only libxml-2.0 and uuid will return something after the upcoming change since they have slightly different include paths.

[0xTim]

@Bo98 awesome, thanks for the help!

[dlbuckley]

Thanks for the help @Bo98

[0xTim]

@Bo98 by the way, is there an easy way for me to test this locally?

[MikeMcQuaid]

@MikeMcQuaid I don't think this comment was really called for 😞 We are just trying to figure out what's going on and have narrowed it down to some change in homebrew. Comments like this push away people who are actively trying to find issues and contribute.

@dlbuckley We're a group of unpaid volunteers working on crucial software for millions of people. Comments like "as its essentially halted production of our product as there doesn't seem to be a way to downgrade homebrew to an older version to continue work" do not contribute to the discussion or finding a solution. We do not consider that a useful contribution to Homebrew.

You can stop Homebrew from autoupdating, Homebrew only upgrades when you run commands and (with a bit of research) you'd be able to figure out that you can roll back any Homebrew change if needed. If you're going to bring your business into it (rather that just talk about the implementation details): you may get feedback on how to avoid future disruption to your business.

@MikeMcQuaid apologies if I came off as accusationary in my initial comment, to be clear I'm not saying it's definitely a Homebrew error - it could well be SwiftPM using something they shouldn't be, but the change for the multiple people reporting the errors was a Homebrew upgrade and I managed to track down the inclusion of the CLTs, which matches the errors, so Homebrew was my first point to raise it

@0xTim No worries, appreciate the apology ❤️. In future it's generally worth debugging "down the stack" i.e. report this to SwiftPM and only file here if they've explicitly said it's a Homebrew issue.

[ydnar]

Hi folks, it looks like this also broke building Go packages that depend on pkg-config for CGO dependencies.

Some sample output from a Go project:

❯ go test ./...
# /usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config --cflags  -- opus opus opus opusfile opus opusfile
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: /pkg-config/bin/pkg-config: No such file or directory
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: exec: /pkg-config/bin/pkg-config: cannot execute: No such file or directory
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: exit status 126

Looks like HOMEBREW_OPT isn’t set. I’d love to help—how best to?

[Bo98]

Why is Go using the superenv shims outside of the Homebrew environment? The shims are strictly only for use inside the superenv.

[Bo98]

@0xTim You could maybe modify and build a version of pkg-config locally yourself. Otherwise, the fix will be out in a couple hours if all goes well.

[ydnar]

Why is Go using the superenv shims outside of the Homebrew environment? The shims are strictly only for use inside the superenv.

Great question. This is on a Homebrew-installed Go (1.14.2). What enables superenv?

[Bo98]

brew sh allows you to be in a superenv. When Homebrew builds things from source (which the CI does to create the bottles), it uses the superenv by default.

[ydnar]

Looks like Homebrew builds Go with superenv:

❯ go env PKG_CONFIG
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config

[Bo98]

Yes, using the superenv while building is correct. I'm just wondering why Go bakes the full pkg-config path into its binaries and how to avoid that. I'll have a look around.

[ydnar]

Yes, using the superenv while building is correct. I'm just wondering why Go bakes the full pkg-config path into its binaries and how to avoid that. I'll have a look around.

I verified that setting PKG_CONFIG at runtime to the correct path allows Go to correctly build cgo packages:

PKG_CONFIG=$(which pkg-config) go test ./...

I think this line is the culprit: https://github.com/Homebrew/brew/pull/7277/files#diff-b7261a4b3eaff4e69f0871ecfd15ca54R115

[Bo98]

I'll try removing that line and generating a new bottle. Hopefully other build systems are able to find the correct pkg-config on their own.

[ydnar]

Thanks. Might be a good idea to upstream this fix into Go, so it doesn’t bake the PKG_CONFIG variable into its binary.

https://github.com/golang/go/blob/d5e1b7ca68e2cc484d9a1517bdc0a9862936a1eb/src/cmd/dist/build.go#L216

Update: filed

[gwynne]

cmake is getting tripped up too:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.15.4
BuildVersion:   19E287
$ xcodebuild -version
Xcode 11.4
Build version 11E146
$ brew info cmake
cmake: stable 3.17.0 (bottled), HEAD
Cross-platform make
https://www.cmake.org/
/usr/local/Cellar/cmake/3.17.0_1 (6,156 files, 58.1MB) *
  Poured from bottle on 2020-03-28 at 11:15:26
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/cmake.rb
<snip>
$ cmake --find-package -DNAME=LibXml2 -DCOMPILER_ID=Clang -DLANGUAGE=C -DMODE=COMPILE
-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/libxml2
-I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/libxml2

Obviously the latter directory doesn't exist - even if I had the CLT installed (which I make a point of not doing), they'd be for 10.15, not 10.14. CMake is intolerant of things specifying nonexistent directories and errors out during configuration.

Manually removing or renaming /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/*/libxml2.0.pc gets rid of the extra include path and everything works again.

[Bo98]

@ydnar I'll have a look. I think other things get baked in as Go installations are not currently relocatable (which sucks for anyone not using /usr/local) so I'll try go through more of them.

@gwynne Did you upgrade to 10.15 recently? The Catalina bottles should not have 10.14 at all.

pkg-config also previously returned /usr/include/libxml2 which is also non-existent. Why did CMake not complain then?

[gwynne]

@Bo98 Oh it did complain before, I'd just added quick one-liner function to my .bash_profile that did for p in /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/*/*.pc; do mv $p $p-DISABLE; done and put up with it on the occasions when I had to invoke it again. And no; I updated to Catalina a long, long time ago, and my cmake bottle was poured much more recently.

[gwynne]

@Bo98 Quick update - I just ran brew upgrade and now it's showing the 10.15 CLT path, which is admittedly slightly more correct in general but still fails out since I neither have nor want the (deprecated!) CLT.

[Bo98]

Given the CLT is the only way to get the correct system headers on Mojave without installing an older Xcode (Mojave system headers were removed in Xcode 11), there's a long long way to go before it's phased out (which I haven't heard any concrete plans of that happening). I would not be surprised if the same situation applies to Catalina when Xcode 12 is out.

Ideally pkg-config would detect whether you have the CLT or Xcode installed but pkg-config is just incapable of doing that. It is designed to work with hardcoded paths only. pkg-config is designed for the Linux eco-system where system headers don't move all over the place like is currently the case on macOS after the removal of /usr/include.

I'm still not entirely sure how CMake breaks if you have an invalid include path. This isn't exactly a rare thing to happen under certain systems. Clang doesn't abort if a -I directory doesn't exist. It's probably also a CMake bug it returns two paths, as it does that even if they match:

$ cmake --find-package -DNAME=LibXml2 -DCOMPILER_ID=Clang -DLANGUAGE=C -DMODE=COMPILE
-I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/libxml2 -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/libxml2

[MikeMcQuaid]

At this point I'm getting increasingly tempted to rip off the band-aid (before the wound has healed) and stop providing these pkg-config files altogether. This is an utter mess that we're basically unable to adequately fix due to the Xcode/CLT and we have way too many parts of the macOS ecosystem relying on undocumented Homebrew internals.

[gwynne]

It breaks because imported targets explicitly validate whether the paths they include make sense:

localhost:Desktop$ mkdir test && cd test
localhost:test$ touch test.c
localhost:test$ cat >CMakeLists.txt <<END
CMAKE_MINIMUM_REQUIRED( VERSION 3.17.0 )
PROJECT( test LANGUAGES C )
FIND_PACKAGE( LibXml2 REQUIRED )
ADD_EXECUTABLE( test test.c )
TARGET_LINK_LIBRARIES( test LibXml2::LibXml2 )
END
localhost:test$ cmake .
-- The C compiler identification is AppleClang 11.0.3.11030032
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found LibXml2: /usr/lib/libxml2.dylib (found version "2.9.4")
-- Configuring done
CMake Error in CMakeLists.txt:
  Imported target "LibXml2::LibXml2" includes non-existent path

    "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/libxml2"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.

CMake Error in CMakeLists.txt:
  Imported target "LibXml2::LibXml2" includes non-existent path

    "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/libxml2"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.

CMake Error in CMakeLists.txt:
  Imported target "LibXml2::LibXml2" includes non-existent path

    "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/libxml2"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.
localhost:test$