railwaycat commented 6 years ago

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

[x] are reporting a bug others will be able to reproduce and not asking a question. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh
[x] ran a brew command and reproduced the problem with multiple formulae? If it's a problem with a single, official formula (not cask) please file this issue at Homebrew/homebrew-core: https://github.com/Homebrew/homebrew-core/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
[x] ran brew update and can still reproduce the problem?
[x] ran brew doctor, fixed all issues and can still reproduce the problem?

[x] ran brew config and brew doctor and included their output with your issue?


% brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: The following directories do not exist: /usr/local/sbin

You should create these directories and change their ownership to your account. sudo mkdir -p /usr/local/sbin sudo chown -R $(whoami) /usr/local/sbin % brew config HOMEBREW_VERSION: 1.7.6-143-gc6acab6 ORIGIN: https://github.com/Homebrew/brew HEAD: c6acab66feb63743fe69b38dab7dc7b90710e838 Last commit: 2 hours ago Core tap ORIGIN: https://github.com/Homebrew/homebrew-core Core tap HEAD: 537f3703aa41c550edbaaf28dd280c6ec929f946 Core tap last commit: 34 minutes ago HOMEBREW_PREFIX: /usr/local HOMEBREW_DEV_CMD_RUN: 1 CPU: octa-core 64-bit kabylake Homebrew Ruby: 2.3.7 => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby Clang: 10.0 build 1000 Git: 2.17.1 => /Applications/Xcode.app/Contents/Developer/usr/bin/git Curl: 7.54.0 => /usr/bin/curl macOS: 10.14-x86_64 CLT: 10.0.0.0.1.1535735448 Xcode: 10.0 XQuartz: 2.7.11 => /opt/X11


To help us debug your issue please explain:
- What you were trying to do (and why)
I tried to build a package which uses `pkg-config` from Homebrew.

- What happened (include command output)
I met an error when build:

fatal error: 'libxml/tree.h' file not found

include <libxml/tree.h>

     ^~~~~~~~~~~~~~~

1 error generated.


It looks like `pkg-config` from Homebrew is not providing the correct include directory of libs shipping with macOS(Xcode and CLI tools). As the release note of Xcode 10, header files which were under `/usr/include` no longer exists and been moved to Xcode: `/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include` and CLI tools: `/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include`.

output:

% which pkg-config /usr/local/bin/pkg-config

% pkg-config --cflags --libs "libxml-2.0 > 2.6.17" -I/usr/include/libxml2 -lxml2

% pkg-config --variable pcfiledir "libxml-2.0 > 2.6.17" /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.14

% cat /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.14/libxml-2.0.pc prefix=/usr exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include modules=1

Name: libXML Version: 2.9.4 Description: libXML library version2. Requires: Libs: -L${libdir} -lxml2 Libs.private: -lz -lpthread -licucore -lm Cflags: -I${includedir}/libxml2


I tested only with `libxml2` but I believe this issue affect all libs which have `pkg-config` file from Homebrew `Library/Homebrew/os/mac/pkgconfig/10.14`, they are: libcurl.pc, libexslt.pc, libxml-2.0.pc, libxslt.pc, sqlite3.pc, zlib.pc.
- What you expected to happen

% pkg-config --cflags --libs "libxml-2.0 > 2.6.17"



should gives the correct path to the header files under Xcode or CLI tools. 

- Step-by-step reproduction instructions (by running `brew` commands)
1. `brew install pkg-config`
2. `pkg-config --cflags --libs "libxml-2.0 > 2.6.17"`

gwynne commented 4 years ago

@MikeMcQuaid I would be 100% behind just ditching the png-config files. It nags at me especially given I have the libxml2 Homebrew formula but can't leverage that one's perfectly good libxml2.0.pc without having to manually specify PKG_CONFIG_PATH or otherwise give cmake extra my-machine-specific hints because it's keg-only (not that that's not for good reasons too...)

MikeMcQuaid commented 4 years ago

@gwynne With respect, we've talked about your views on the CLT in this thread (and in Slack) at length and we're now at the point where we're talking about your personal configuration. I think it's best at this point for you (or anyone else) to provide PRs for debate rather than issue comments, thanks.

Bo98 commented 4 years ago

The SwiftPM situation should hopefully be sorted now. Please run brew upgrade pkg-config.

0xTim commented 4 years ago

Thanks for the quick turnaround @Bo98 and @MikeMcQuaid !

elv-peter commented 4 years ago

Hi folks, it looks like this also broke building Go packages that depend on pkg-config for CGO dependencies.

Some sample output from a Go project:
❯ go test ./...
# /usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config --cflags  -- opus opus opus opusfile opus opusfile
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: /pkg-config/bin/pkg-config: No such file or directory
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: exec: /pkg-config/bin/pkg-config: cannot execute: No such file or directory
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: exit status 126
Looks like HOMEBREW_OPT isn’t set. I’d love to help—how best to?

I don't know what is going on, but export HOMEBREW_OPT=/usr/local/opt in ~/.bash_profile worked for me

Bo98 commented 4 years ago

Run brew upgrade go. I've just pushed new Go bottles.

poenik commented 4 years ago

still not working for me

Bo98 commented 4 years ago

I'll need more information @poenik. With the latest pkg-config, pkg-config --cflags zlib returns nothing which is exactly the same as it was last week.

poenik commented 4 years ago

@Bo98, I have run the brew upgrade pkg-configcommand, it updated succesfully, but the errors still showed up when building. Ultimately, they were solved after following a suggestion from Vapor forum, to delete the /Library/Developer/CommandLineTools folder, as working in Xcode already, it was not needed. This solved the redefinition of modules issue.

Bo98 commented 4 years ago

Looks like there's a significant issue with the way SwiftPM handles pkg-config. It incorrectly handles *.pc files directly so that PKG_CONFIG_SYSTEM_INCLUDE_PATH (be it the environment variable or the baked in default) is not appropriately applied. It attempts to mimic what pkg-config does but this implementation is incomplete: https://github.com/apple/swift-package-manager/blob/d433df8bfca5e3df498b3300c2b6aee222154cd4/Sources/PackageLoading/Target%2BPkgConfig.swift#L244

The SwiftPM issue has fatal effects on macOS due to systems headers no longer being at /usr/include since Mojave (SwiftPM only filters /usr/include still).

This isn't however something macOS specific. There are multiarch Linux distros that rely on this. There's also other variables commonly used in places on Linux like PKG_CONFIG_ALLOW_SYSTEM_LIBS. SwiftPM fails to recognise any of this.

There is a general issue covering the topic at https://bugs.swift.org/browse/SR-6317, filed internally as rdar://problem/40235197.

I'll look into deleting cflags entirely until the Swift issue is fixed.

Bo98 commented 4 years ago

I think all issues mentioned here will be fixed in Homebrew 2.2.13.

0xTim commented 4 years ago

Thanks for all the work @Bo98 🎉

pgrenaud commented 4 years ago

I'm sorry to be the one that brings bad news, but I think I have an issue related to this. Since yesterday, I'm unable to build the imagick PECL extension (for PHP) that I used to be able to build without any issue. It's still unclear to me if the macOS update (or supplemental update), this Homebrew update, PHP, PECL or imagick is this cause of this issue. But everything seems to point to pkg-config.

I use a script to provision and keep up-to-date our macOS development environment and I always run it periodically in macOS VMs to be sure that nothing's broken. Here's what I got this morning when the script ran yes '' | pecl install --force imagick for PHP 7.4:

downloading imagick-3.4.4.tgz ...
Starting to download imagick-3.4.4.tgz (253,434 bytes)
.....................................................done: 253,434 bytes
19 source files, building
running: phpize
Configuring for:
PHP Api Version:         20190902
Zend Module Api No:      20190902
Zend Extension Api No:   320190902
Please provide the prefix of ImageMagick installation [autodetect] : 
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in Builder.php on line 405
PHP Warning:  Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /usr/local/Cellar/php/7.4.4/share/php/pear/PEAR/Builder.php on line 405

Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /usr/local/Cellar/php/7.4.4/share/php/pear/PEAR/Builder.php on line 405
building in /private/tmp/pear/temp/pear-build-exolnetnl3Foi/imagick-3.4.4
running: /private/tmp/pear/temp/imagick/configure --with-php-config=/usr/local/opt/php/bin/php-config --with-imagick
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for a sed that does not truncate output... /usr/bin/sed
checking for pkg-config... /usr/local/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for cc... cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether cc accepts -g... yes
checking for cc option to accept ISO C89... none needed
checking how to run the C preprocessor... cc -E
checking for icc... no
checking for suncc... no
checking for system library directory... lib
checking if compiler supports -R... no
checking if compiler supports -Wl,-rpath,... yes
checking build system type... x86_64-apple-darwin19.4.0
checking host system type... x86_64-apple-darwin19.4.0
checking target system type... x86_64-apple-darwin19.4.0
checking for PHP prefix... /usr/local/Cellar/php/7.4.4
checking for PHP includes... -I/usr/local/Cellar/php/7.4.4/include/php -I/usr/local/Cellar/php/7.4.4/include/php/main -I/usr/local/Cellar/php/7.4.4/include/php/TSRM -I/usr/local/Cellar/php/7.4.4/include/php/Zend -I/usr/local/Cellar/php/7.4.4/include/php/ext -I/usr/local/Cellar/php/7.4.4/include/php/ext/date/lib
checking for PHP extension directory... /usr/local/Cellar/php/7.4.4/pecl/20190902
checking for PHP installed headers prefix... /usr/local/Cellar/php/7.4.4/include/php
checking if debug is enabled... no
checking if zts is enabled... no
checking for gawk... no
checking for nawk... no
checking for awk... awk
checking if awk is broken... no
checking whether to enable the imagick extension... yes, shared
checking for pkg-config... /usr/local/bin/pkg-config
checking ImageMagick MagickWand API configuration program... checking Testing /usr/local/bin/MagickWand-config... It exists
found in /usr/local/bin/MagickWand-config
checking if ImageMagick version is at least 6.2.4... found version 7.0.10 Q16 HDRI
checking for MagickWand.h or magick-wand.h header... /usr/local/Cellar/imagemagick/7.0.10-6/include/ImageMagick-7/MagickWand/MagickWand.h
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: /pkg-config/bin/pkg-config: No such file or directory
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: exec: /pkg-config/bin/pkg-config: cannot execute: No such file or directory
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: /pkg-config/bin/pkg-config: No such file or directory
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: exec: /pkg-config/bin/pkg-config: cannot execute: No such file or directory
checking PHP version is at least 5.3.2... yes. found 7.4.4
libs

checking for MagickGetVersion... no
checking for __MagickGetVersion... no
checking for a sed that does not truncate output... /usr/bin/sed
checking for ld used by cc... /Library/Developer/CommandLineTools/usr/bin/ld
checking if the linker (/Library/Developer/CommandLineTools/usr/bin/ld) is GNU ld... no
checking for /Library/Developer/CommandLineTools/usr/bin/ld option to reload object files... -r
checking for BSD-compatible nm... /usr/bin/nm -B
checking whether ln -s works... yes
checking how to recognize dependent libraries... pass_all
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking dlfcn.h usability... yes
checking dlfcn.h presence... yes
checking for dlfcn.h... yes
checking the maximum length of command line arguments... 196608
checking command to parse /usr/bin/nm -B output from cc object... ok
checking for objdir... .libs
checking for ar... ar
checking for ranlib... ranlib
checking for strip... strip
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for -single_module linker flag... yes
checking for -exported_symbols_list linker flag... yes
checking if cc supports -fno-rtti -fno-exceptions... yes
checking for cc option to produce PIC... -fno-common
checking if cc PIC flag -fno-common works... yes
checking if cc static flag -static works... no
checking if cc supports -c -o file.o... yes
checking whether the cc linker (/Library/Developer/CommandLineTools/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin19.4.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no

creating libtool
appending configuration tag "CXX" to libtool
configure: patching config.h.in
configure: creating ./config.status
config.status: creating config.h
running: make
/bin/sh /private/tmp/pear/temp/pear-build-exolnetnl3Foi/imagick-3.4.4/libtool --mode=compile cc  -I. -I/private/tmp/pear/temp/imagick -DPHP_ATOM_INC -I/private/tmp/pear/temp/pear-build-exolnetnl3Foi/imagick-3.4.4/include -I/private/tmp/pear/temp/pear-build-exolnetnl3Foi/imagick-3.4.4/main -I/private/tmp/pear/temp/imagick -I/usr/local/Cellar/php/7.4.4/include/php -I/usr/local/Cellar/php/7.4.4/include/php/main -I/usr/local/Cellar/php/7.4.4/include/php/TSRM -I/usr/local/Cellar/php/7.4.4/include/php/Zend -I/usr/local/Cellar/php/7.4.4/include/php/ext -I/usr/local/Cellar/php/7.4.4/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /private/tmp/pear/temp/imagick/imagick_file.c -o imagick_file.lo
mkdir .libs
 cc -I. -I/private/tmp/pear/temp/imagick -DPHP_ATOM_INC -I/private/tmp/pear/temp/pear-build-exolnetnl3Foi/imagick-3.4.4/include -I/private/tmp/pear/temp/pear-build-exolnetnl3Foi/imagick-3.4.4/main -I/private/tmp/pear/temp/imagick -I/usr/local/Cellar/php/7.4.4/include/php -I/usr/local/Cellar/php/7.4.4/include/php/main -I/usr/local/Cellar/php/7.4.4/include/php/TSRM -I/usr/local/Cellar/php/7.4.4/include/php/Zend -I/usr/local/Cellar/php/7.4.4/include/php/ext -I/usr/local/Cellar/php/7.4.4/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /private/tmp/pear/temp/imagick/imagick_file.c  -fno-common -DPIC -o .libs/imagick_file.o
In file included from /private/tmp/pear/temp/imagick/imagick_file.c:22:
In file included from /private/tmp/pear/temp/imagick/php_imagick_file.h:24:
/private/tmp/pear/temp/imagick/php_imagick_defs.h:25:12: fatal error: 'MagickWand/MagickWand.h' file not found
#  include <MagickWand/MagickWand.h>
           ^~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [imagick_file.lo] Error 1
ERROR: `make' failed

Of course, I though at first that this was an issue with imagick. But reinstalling it does not fix the issue. I have the same issue with PHP 7.2, 7.3 and 7.4, all installed with Homebrew. So far, I've reinstalled php@7.2, php@7.3, php, imagemagick and pkg-config. Homebrew (2.2.13-28-ga348a14) and macOS Catalina (10.15.4) are up-to-date.

Then, I found this blog post linked from this stackoverflow answer. Following the advice, I ran yes '' | C_INCLUDE_PATH=/usr/local/Cellar/imagemagick/7.0.10-6/include/ImageMagick-7 pecl install --force imagick to try installing the imagick extension again, and it worked!

My knowledge is limited, but considering the line /usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: /pkg-config/bin/pkg-config: No such file or directory from the building output above, and considering that I'm able to build imagick if I specify the include path, I assume that the issue is related to pkg-config, right?

Bo98 commented 4 years ago

That's a ImageMagick-specific problem. Their build scripts bake in several paths based on the build environment which is not the same as the runtime environment. I'll see if there's a way to workaround that.

mkalvas commented 4 years ago

I'm having the same issue as @pgrenaud with the rmagick gem (which builds native extrensions with imagemagick) in a ruby on rails env. i.e.

/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: /pkg-config/bin/pkg-config: No such file or directory

Which results in rmagick failing out as Can't find the ImageMagick library or one of the dependent libraries. even though they are all there and installed correctly. Also, notably, the workaround for building through adding the C_INCLUDE_PATH and/or PKG_CONFIG_PATH doesn't work (seemingly only on this version of homebrew?).

During my investigation, I've found that however the .../super/pkg-config file is being run $HOMEBREW_OPT isn't available which is why we're getting /pkg-config as the root of the executable path. Adding HOMEBREW_OPT to the command gets it working.

e.g. the working build command

C_INCLUDE_PATH=/usr/local/Cellar/imagemagick@6/6.9.11-6/include/ImageMagick-6 PKG_CONFIG_PATH=/usr/local/Cellar/imagemagick@6/6.9.11-6/lib/pkgconfig HOMEBREW_OPT=/usr/local/opt gem install rmagick

I just wanted to add my experience and info in case it was helpful for other people here or googling in the future.

Edit: Also, I have all of these exported and everything in my .zshrc, but only including them for the command is working.

Bo98 commented 4 years ago

@pgrenaud @mkalvas Run brew upgade imagemagick/brew upgrade imagemagick@6 and try again. Should work now.

pgrenaud commented 4 years ago

Wow, thanks for the quick fix @Bo98 ! 👏 I confirm that everything is working as expected on my side! 👌

MikeMcQuaid commented 4 years ago

This long-running issue was (finally) fixed by the incredible @Bo98. Thanks for all your work @Bo98 👏

artob commented 4 years ago

I think all issues mentioned here will be fixed in Homebrew 2.2.13.

@Bo98 I'm afraid the Go (specifically, cgo) issue previously encountered also by @ydnar @elv-peter and @poenik still persists with the latest.

If I've understood this long thread correctly, the problem arises because Go hardcodes the pkg-config shim path into its binaries, that shim needs HOMEBREW_OPT set, and Go upstream told us there's nothing to fix on their end?

In any case, here's a minimal reproduction of the problem with Homebrew 2.2.14, pkg-config 0.29.2_3, and go 1.14.2_1 on macOS 10.14:

$ cat main.go
package main

// #cgo pkg-config: zlib
// extern const char* zlibVersion();
import "C"
import "fmt"

func main() {
    fmt.Println(C.GoString(C.zlibVersion()))
}

Build failure:

$ go run main.go
# /usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config --cflags  -- zlib
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: /pkg-config/bin/pkg-config: No such file or directory
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: line 6: exec: /pkg-config/bin/pkg-config: cannot execute: No such file or directory
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/pkg-config: exit status 126

Build workaround (kudos to @ydnar):

$ PKG_CONFIG=$(which pkg-config) go run main.go
1.2.11

(Alternative workaround: set HOMEBREW_OPT as demonstrated in a couple of previous replies.)

Bo98 commented 4 years ago

I can't reproduce:

$ go run main.go
1.2.11

If you build Go from source rather than install from a bottle, make sure it was done so recently (and brew reinstall go if not).

artob commented 4 years ago

@Bo98 Thanks. I haven't purposely built Go from source, but you are correct, it does look like it was built from source:

$ brew info go
go: stable 1.14.2, HEAD
Open source programming language to build simple/reliable/efficient software
https://golang.org
/usr/local/Homebrew/Cellar/go/1.14.2_1 (9,440 files, 424.4MB) *
  Built from source on 2020-04-11 at 13:46:44

brew reinstall go just wants to rebuild it. Probably would be preferable to use the bottle; I'll look later to see how I can force the bottle installation.

artob commented 4 years ago

I can't reproduce:
$ go run main.go
1.2.11
If you build Go from source rather than install from a bottle, make sure it was done so recently (and brew reinstall go if not).

@Bo98 I confirm that brew reinstall go did solve the issue, and that the problem is no longer reproducible on my laptop.

Homebrew / brew

pkg-config files on >=10.14 assume headers in /usr/include #5068

include <libxml/tree.h>