search

Homebrew / homebrew-core

🍻 Default formulae for the missing package manager for macOS (or Linux)
https://brew.sh
BSD 2-Clause "Simplified" License
13.62k stars 12.36k forks source link

libxml2 install path #56402

Closed asmichel closed 4 years ago

asmichel commented 4 years ago

What you were trying to do (and why)

Install libxml2 and then run cpanm XML::LibXML

What happened (include command output)

cpanm XML::LibXML failed, giving the error

Devel.xs:28:10: fatal error: 'libxml/xmlmemory.h' file not found
#include <libxml/xmlmemory.h>
         ^~~~~~~~~~~~~~~~~~~~

even though I had already set

export LDFLAGS="-L/usr/local/opt/libxml2/lib"
export CPPFLAGS="-I/usr/local/opt/libxml2/include/"

as suggested by Homebrew during the installation of libxml2. After some digging I found that the suggested CPPFLAGS was incorrect, and should instead be

export CPPFLAGS="-I/usr/local/opt/libxml2/include/libxml2/"

Since clang seems to ignore LDFLAGS and CPPFLAGS, I ended up setting

export C_INCLUDE_PATH="/usr/local/opt/libxml2/include/libxml2/"
export CPLUS_INCLUDE_PATH="/usr/local/opt/libxml2/include/libxml2/"
export LIBRARY_PATH="/usr/local/opt/libxml2/lib"

which finally worked.

Why does the homebrew package use this directory structure for the header files? At the very least, the package should be updated to tell users to set the environment variables correctly.

This was not the end of the issue though. After successfully arranging for libxml2 to be found by clang, cpanm XML::LibXML still failed to install, giving this error

Warning: XML::LibXML compiled against libxml2 20910, but runtime libxml2 is older 20904

#   Failed test 'LIBXML__VERSION == LIBXML_RUNTIME_VERSION'
#   at t/01basic.t line 18.
#          got: '20910'
#     expected: '20904'
# DO NOT REPORT THIS FAILURE: Your setup of library paths is incorrect!
# 
# 
# Compiled against libxml2 version: 20910
# Running libxml2 version:          20904
#

Thankfully, cpanm XML::LibXML --force succeeded in installing the package, but I'm still worried about this error. Should I just ignore it, since Homebrew deliberately preserves older versions of libraries packaged with MacOS? What is the "runtime" libxml2 version?

SMillerDev commented 4 years ago

The reason you had to set those paths is because homebrew tries to avoid overloading the macOS provided tools. Is there any reason you specifically need the homebrew version?

asmichel commented 4 years ago

Without the Homebrew package, cpanm XML::LibXML would fail to install. I believe that the prepackaged version that comes with MacOS doesn't have the header files (otherwise, why would cpanm fail?)

asmichel commented 4 years ago

Also, I understand Homebrew's general philosophy of respecting the system tools. That was not my main problem; I was only unclear on why

export CPPFLAGS="-I/usr/local/opt/libxml2/include/"

failed and 

export CPPFLAGS="-I/usr/local/opt/libxml2/include/libxml2/"

succeeded ie why I needed an extra /libxml2/ after the /include/, as opposed to the path given in the homebrew package installation instructions:

libxml2 is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have libxml2 first in your PATH run:
  echo 'export PATH="/usr/local/opt/libxml2/bin:$PATH"' >> /Users/asmichel/.bash_profile

For compilers to find libxml2 you may need to set:
  export LDFLAGS="-L/usr/local/opt/libxml2/lib"
  export CPPFLAGS="-I/usr/local/opt/libxml2/include"
follower commented 4 years ago

TL;DR: Try using pkg-config? Maybe there's a bug related to recent changes?

I don't see any information about what version of OS/brew you're running but, I note that libxml2 seems to be a "special case" for a number of reasons related to there also being a pre-installed system version.

Recent libxml2-related changes

With that in mind you may be being affected by these changes or bugs in these changes:

Which prompts the question...

How are caveats generated?

As far as I can tell the caveats are generated by this code:

Which seem to refer to the default values generated via the base Formula class for both include & opt_include:

(Which makes me start to wonder if the suggested flags ever included the required libxml2 suffix...)

Why am I here?

I ended up here because I was trying to understand why some formulas (e.g. libffi) seemed to include a pkg-config-related caveat and others (e.g. libxml2) didn't.

And, what I discovered (I think) is that the pkg-config related caveat isn't related to the formula but rather whether pkg-config is installed (well, installed by Homebrew, maybe, it got a bit confusing :D ) which was...unexpected.

Anyway, I was going to suggest you try installing/using pkg-config to see if it gave you the correct paths--both to assist you & maybe provide some insight into the LDFLAGS/CPPFLAGS issue.

More about conditional pkg-config caveats

As mentioned above, it turns out that pkg-config caveats are only displayed in certain circumstances controlled by:

Primarily this line here...:

      if which("pkg-config", ENV["HOMEBREW_PATH"]) &&
         ((f.lib/"pkgconfig").directory? || (f.share/"pkgconfig").directory?)

...which as I understand it (I'm not a Rubyist) looks for pkg-config in the "filtered" HOMEBREW_PATH. And only displays the caveat if it's found. (But then shouldn't libxml2 get one too?)

It used to be that it checked the entire environment (see original: https://github.com/Homebrew/brew/pull/917, https://github.com/Homebrew/brew/commit/2394d6f2d8d00dfe6dd3824de678d3ca44e9ae57; see after https://github.com/Homebrew/brew/issues/932: https://github.com/Homebrew/brew/pull/3552 & https://github.com/Homebrew/brew/commit/d70a406fe7a8820634adf217c9962d613ffe4cc0), which presumably would cover the case where pkg-config was installed by non-Homebrew means.

While I understand the desire to not fatigue people with caveat messages, I'm of the opinion that:

Added symlink complications

The situation is additionally complicated by the fact that non-keg formulas are sym-linked so if someone goes looking for a .pc file, they'll find e.g. via:

mdfind -name libffi.pc -onlyin /

a result like:

/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/libffi.pc
/opt/pkg/lib/pkgconfig/libffi.pc

And promptly use the version-specific /usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/libffi.pc (as I initially did) instead of the more robust version independent /usr/local/opt/libffi/lib/pkgconfig/libffi.pc (or /usr/local/opt/libffi/lib/pkgconfig as appropriate).

In fact, it was only my desire to use PKG_CONFIG_LIBDIR/PKG_CONFIG_PATH that prompted me to dig deeper & find the version-independent symlink path because I thought "surely I shouldn't be hard-coding a version number, so there must be some way to get a version-independent path", so ran brew info libffi and discovered the caveat.

(I was considering replacing my use of a pkgsrc installed libffi with a brew installed one as a default in a Makefile because I figured the brew case was more common--and I was specifically wanting to static link with libffi.a--and was using mdfind to find an alternative, rather than the brew CLI. There just happened to be a Homebrew installed libffi lying around. :D )

The impact of the symlink aspect is that there's a bunch of people on the internet telling people to use a version-specific path to solve their problem instead of the version independent one:

Semi-related link:

Anyway, perhaps one of these pointers will help in your situation.

Edits:

Bo98 commented 4 years ago

The CPPFLAGS is just a general recommendation for keg-only formula. If libxml2 wasn't keg-only, you wouldn't have been given any assistance at all. What you're seeing is just because libxml2 is unique in having an extra subdirectory that is often needed in the -I flag.

For libxml2, I highly recommend either using pkg-config (PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig") or even setting the path (PATH="/usr/local/opt/libxml2/bin:$PATH") if you want to use xml2-config. Both are in the caveats. Most software rely on one or the other to find libxml2 and I recommend using them over setting CPPFLAGS. This follows what upstream recommends.

Bo98 commented 4 years ago

And, what I discovered (I think) is that the pkg-config related caveat isn't related to the formula but rather whether pkg-config is installed (well, installed by Homebrew, maybe, it got a bit confusing :D ) which was...unexpected.

It's a bit of both. It checks if pkg-config is installed and the formula provides pkg-config files.

Feel free to open a Homebrew/brew pull request to remove the pkg-config installed check if you think it is clearer that way. That's the best way to start a conversation on the matter.

asmichel commented 4 years ago

In light of recent comments I went ahead and checked if pkg-config was installed, and I found that it wasn't there. So it seems that when I initially ran brew install libxml2, homebrew didn't install pkg-config as a dependency. Is this expected behavior, or is my system just really messed up for some reason (ie pkg-config was supposed to be installed a long time ago but for some reason wasn't)?

So just to make sure I understand what you guys have said—I should remove libxml2, install pkg-config, and then install libxml2 again, which should properly configure itself this time. Is this right?

Bo98 commented 4 years ago

Is this expected behavior

Yes because pkg-config isn’t strictly required. Just encouraged for development.

I should remove libxml2, install pkg-config, and then install libxml2 again

This is not necessary.

In the case of caveats, you can view them again via brew info libxml2.

asmichel commented 4 years ago

Unfortunately, installing pkg-config and running

export PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig"

as per the caveat that shows up in brew info libxml2 doesn't seem to help the cpan XML::LibXML makefile find the library (when I looked through it, I didn't see any calls to pkg-config anywhere).

However, I went and looked through the build log that cpanm XML::LibXML generated and it seems that the build is using xml2-config, not pkg-config. I figured this out because I kept seeing references to the old library that comes with the macOS command-line tools. How do I get xml2-config to see the new library? I already did as the caveat suggested and set

export PATH="/usr/local/opt/libxml2/bin:$PATH"

which unfortunately doesn't work. Running xml2-config --help prints

Usage: xml2-config [OPTION]

Known values for OPTION are:

  --prefix=DIR      change libxml prefix [default /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr]
  --exec-prefix=DIR change libxml exec prefix [default /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr]
  --libs        print library linking information
  --cflags      print pre-processor and compiler flags
  --modules     module support enabled
  --help        display this help and exit
  --version     output version information

and I have no idea how to change these defaults.

follower commented 4 years ago

However, I went and looked through the build log that cpanm XML::LibXML generated and it seems that the build is using xml2-config, not pkg-config. I figured this out because I kept seeing references to the old library that comes with the macOS command-line tools. How do I get xml2-config to see the new library?

Hmmm, I do vaguely remember reading something about projects essentially building/having a "custom" pkg-config which uses a unique prefix.

I suspect that's probably a question to ask the libxml2 project but perhaps the tool reads the same environment variables but with a replacement for the PKG prefix (e.g. XML2)?

shlomif commented 4 years ago

Unfortunately, installing pkg-config and running

export PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig"

as per the caveat that shows up in brew info libxml2 doesn't seem to help the cpan XML::LibXML makefile find the library (when I looked through it, I didn't see any calls to pkg-config anywhere).

However, I went and looked through the build log that cpanm XML::LibXML generated and it seems that the build is using xml2-config, not pkg-config. I figured this out because I kept seeing references to the old library that comes with the macOS command-line tools. How do I get xml2-config to see the new library? I already did as the caveat suggested and set

export PATH="/usr/local/opt/libxml2/bin:$PATH"

If you are using zsh/etc. you may need to run rehash after that command.

which unfortunately doesn't work. Running xml2-config --help prints

Usage: xml2-config [OPTION]

Known values for OPTION are:

  --prefix=DIR        change libxml prefix [default /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr]
  --exec-prefix=DIR   change libxml exec prefix [default /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr]
  --libs      print library linking information
  --cflags        print pre-processor and compiler flags
  --modules       module support enabled
  --help      display this help and exit
  --version       output version information

and I have no idea how to change these defaults.

asmichel commented 4 years ago

@shlomif I'm not using zsh, just bash.

Edit: it seems that the command is hash -r. I’ll run it then then report back.

asmichel commented 4 years ago

I created a test file

#include <stdio.h>
#include <libxml/xmlmemory.h>

main()
{
    printf("hello, world\n");
}

and after running hash -r, this compiles

cc $(xml2-config --libs --cflags) main.c

But, somehow, cpan XML::LibXML still fails, with the same error about not being able to find the header files (because it's still looking in the default directories). For some reason, it's not using the copy of xml2-config that bash is using.

The only thing that I can think of is that LibXML hardcodes a reference to /usr/bin/xml2-config, which uses the incorrect library directory. The version of xml2-config that it should be using is the one located in /usr/local/opt/libxml2/bin/xml2-config, and it's the one that bash executes when you run xml2-config.

shlomif commented 4 years ago

@alexmichelorg : recent CPAN versions of XML::LibXML depend on Alien-Libxml2 for finding libxml2 so perhaps try reinstalling it (= A-libxml2)

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.