search

Homebrew / homebrew-core

🍻 Default formulae for the missing package manager for macOS (or Linux)
https://brew.sh
BSD 2-Clause "Simplified" License
13.75k stars 12.43k forks source link

`hdf5@1.14.3` package is broken: `h5cc -show` does not work #159691

Closed BenWibking closed 9 months ago

BenWibking commented 9 months ago

Edit: This was caused by https://github.com/Homebrew/homebrew-core/pull/159165. That PR should be reverted.

brew gist-logs <formula> link OR brew config AND brew doctor output

HOMEBREW_VERSION: 4.2.3
ORIGIN: https://github.com/Homebrew/brew
HEAD: b3751bca8c4a3c76107d5aa3b75b81db93cf4bb6
Last commit: 4 days ago
Core tap JSON: 11 Jan 18:45 UTC
Core cask tap JSON: 11 Jan 18:45 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.hhJX4CM40W/org.xquartz:0
HOMEBREW_EDITOR: /opt/homebrew/bin/emacs
HOMEBREW_MAKE_JOBS: 10
Homebrew Ruby: 3.1.4 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 15.0.0 build 1500
Git: 2.43.0 => /opt/homebrew/bin/git
Curl: 8.4.0 => /usr/bin/curl
macOS: 14.2.1-arm64
CLT: 15.1.0.0.1.1700200546
Xcode: 15.2
Rosetta 2: false

Your system is ready to brew.

Verification

What were you trying to do (and why)?

hdf5 provides h5cc, which is a command-line utility that is used by build systems (e.g., CMake) to determine what compiler flags to use to link with hdf5.

This is done with h5cc -show (https://manpages.debian.org/testing/hdf5-helpers/h5cc.1.en.html#show).

What happened (include all command output)?

$ h5cc -show
/opt/homebrew/h5cc
dir is /opt/homebrew
clang: error: unknown argument: '-show'

What did you expect to happen?

It should list the compiler flags needed to link with hdf5.

Note that h5cc -showconfig works, but this cannot be parsed by build systems:

$ h5cc -showconfig
/opt/homebrew/h5cc
dir is /opt/homebrew
        SUMMARY OF THE HDF5 CONFIGURATION
        =================================

General Information:
-------------------
                   HDF5 Version: 1.14.3
                  Configured on: 2023-10-28
                  Configured by: Unix Makefiles
                    Host system: Darwin-23.2.0
              Uname information: Darwin
                       Byte sex: little-endian
             Installation point: /opt/homebrew/Cellar/hdf5/1.14.3

Compiling Options:
------------------
                     Build Mode: Release
              Debugging Symbols: OFF
                        Asserts: OFF
                      Profiling: OFF
             Optimization Level: OFF

Linking Options:
----------------
                      Libraries:
  Statically Linked Executables: OFF
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:
                Extra libraries: m;dl
                       Archiver: /usr/bin/ar
                       AR_FLAGS:
                         Ranlib: /usr/bin/ranlib

Languages:
----------
                              C: YES
                     C Compiler: clang 15.0.0.15000100
                       CPPFLAGS:
                    H5_CPPFLAGS:
                    AM_CPPFLAGS:
                         CFLAGS:  -std=c99
                      H5_CFLAGS: -Wall;-Warray-bounds;-Wcast-qual;-Wconversion;-Wdouble-promotion;-Wextra;-Wformat=2;-Wframe-larger-than=16384;-Wimplicit-fallthrough;-Wnull-dereference;-Wunused-const-variable;-Wwrite-strings;-Wpedantic;-Wvolatile-register-var;-Wno-c++-compat;-Wbad-function-cast;-Wimplicit-function-declaration;-Wincompatible-pointer-types;-Wmissing-declarations;-Wpacked;-Wshadow;-Wswitch;-Wno-error=incompatible-pointer-types-discards-qualifiers;-Wunused-function;-Wunused-variable;-Wunused-parameter;-Wcast-align;-Wformat;-Wno-missing-noreturn
                      AM_CFLAGS:
               Shared C Library: YES
               Static C Library: YES

                        Fortran: ON
               Fortran Compiler: /opt/homebrew/bin/gfortran 13.2.0
                  Fortran Flags:  -fdiagnostics-urls=never -fno-diagnostics-color
               H5 Fortran Flags: -Waliasing;-Wall;-Wcharacter-truncation;-Wextra;-Wimplicit-interface;-Wsurprising;-Wunderflow;-pedantic;-Wno-unused-dummy-argument;-ffree-form;-fimplicit-none;-std=f2008;-Wintrinsics-std;-Wimplicit-procedure;-Wreal-q-constant;-Wfunction-elimination;-Wrealloc-lhs;-Wrealloc-lhs-all;-Wno-c-binding-type;-Wno-array-temporaries;-Winteger-division;-Wfrontend-loop-interchange
               AM Fortran Flags:
         Shared Fortran Library: YES
         Static Fortran Library: YES
               Module Directory: /tmp/hdf5-20240109-4062-6ucskv/hdf5-1.14.3/build/mod

                            C++: ON
                   C++ Compiler: clang++ 15.0.0.15000100
                      C++ Flags:
                   H5 C++ Flags: -Wall;-Warray-bounds;-Wcast-qual;-Wconversion;-Wdouble-promotion;-Wextra;-Wformat=2;-Wframe-larger-than=16384;-Wimplicit-fallthrough;-Wnull-dereference;-Wunused-const-variable;-Wwrite-strings;-Wpedantic;-Wvolatile-register-var;-Wno-c++-compat;-Wno-missing-noreturn
                   AM C++ Flags:
             Shared C++ Library: YES
             Static C++ Library: YES

                           JAVA: OFF
                  JAVA Compiler:

Features:
---------
                     Parallel HDF5: OFF
  Parallel Filtered Dataset Writes:
                Large Parallel I/O:
                High-level library: ON
Dimension scales w/ new references:
                  Build HDF5 Tests: OFF
                  Build HDF5 Tools: ON
                   Build GIF Tools: OFF
                      Threadsafety: OFF
               Default API mapping: v114
    With deprecated public symbols: ON
            I/O filters (external):  DEFLATE DECODE ENCODE
                     Map (H5M) API:
                        Direct VFD:
                        Mirror VFD:
                     Subfiling VFD:
                (Read-Only) S3 VFD:
              (Read-Only) HDFS VFD:
    Packages w/ extra debug output:
                       API Tracing: OFF
              Using memory checker: OFF
            Function Stack Tracing: OFF
                  Use file locking: best-effort
         Strict File Format Checks: OFF
      Optimization Instrumentation:

Step-by-step reproduction instructions (by running brew commands)

$ brew reinstall hdf5
==> Downloading https://ghcr.io/v2/homebrew/core/hdf5/manifests/1.14.3-1
Already downloaded: /Users/benwibking/Library/Caches/Homebrew/downloads/7017517953cd6c5badbb1760f557c406ad3d87f77e43c40ae660c2733bf5c7e4--hdf5-1.14.3-1.bottle_manifest.json
==> Fetching hdf5
==> Downloading https://ghcr.io/v2/homebrew/core/hdf5/blobs/sha256:e6638bc9a427fea8b0780f109435ef0cab0e6283a1c37f57b9f53e4d
Already downloaded: /Users/benwibking/Library/Caches/Homebrew/downloads/36a82074d498645be3ca06bfc60b7c4cd5fa3354c5936be60c25df4d37f54a5c--hdf5--1.14.3.arm64_sonoma.bottle.1.tar.gz
==> Reinstalling hdf5
==> Pouring hdf5--1.14.3.arm64_sonoma.bottle.1.tar.gz
🍺  /opt/homebrew/Cellar/hdf5/1.14.3: 289 files, 77.7MB
==> Running `brew cleanup hdf5`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
(base) ➜  ~ h5cc -show
/opt/homebrew/h5cc
dir is /opt/homebrew
clang: error: unknown argument: '-show'
BenWibking commented 9 months ago

hdf5@1.10 works correctly:

$ h5cc -show
clang -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/Cellar/hdf5@1.10/1.10.11/lib /opt/homebrew/Cellar/hdf5@1.10/1.10.11/lib/libhdf5_hl.a /opt/homebrew/Cellar/hdf5@1.10/1.10.11/lib/libhdf5.a -L/opt/homebrew/opt/libaec/lib -Wl,-ld_classic -lsz -lz -ldl -lm
$ h5cc -showconfig
        SUMMARY OF THE HDF5 CONFIGURATION
        =================================

General Information:
-------------------
                   HDF5 Version: 1.10.11
                  Configured on: Wed Sep 27 22:08:03 UTC 2023
                  Configured by: brew@Ventura-arm64.local
                    Host system: aarch64-apple-darwin23.0.0
              Uname information: Darwin Ventura-arm64.local 23.0.0 Darwin Kernel Version 23.0.0: Thu Aug 17 21:24:15 PDT 2023; root:xnu-10002.1.11~3/RELEASE_ARM64_VMAPPLE arm64
                       Byte sex: little-endian
             Installation point: /opt/homebrew/Cellar/hdf5@1.10/1.10.11

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables:
                        LDFLAGS: -Wl,-ld_classic
                     H5_LDFLAGS:
                     AM_LDFLAGS:  -L/opt/homebrew/opt/libaec/lib
                Extra libraries: -lsz -lz -ldl -lm
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: clang ( Apple clang version 15.0.0 )
                       CPPFLAGS:
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS:  -I/opt/homebrew/opt/libaec/include
                        C Flags:
                     H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3 @H5_ECFLAGS@
                     AM C Flags:
               Shared C Library: yes
               Static C Library: yes

                        Fortran: yes
               Fortran Compiler: /opt/homebrew/opt/gcc/bin/gfortran ( GNU Fortran (Homebrew GCC 13.2.0) 13.2.0)
                  Fortran Flags:
               H5 Fortran Flags:  -std=f2008  -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow -pedantic -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type -Winteger-division -Wfrontend-loop-interchange  -fdiagnostics-urls=never -fno-diagnostics-color -s  -Wno-unused-dummy-argument -Wno-array-temporaries -O3
               AM Fortran Flags:
         Shared Fortran Library: yes
         Static Fortran Library: yes
               Module Directory: ${includedir}

                            C++: yes
                   C++ Compiler: clang++ ( Apple clang version 15.0.0 )
                      C++ Flags:
                   H5 C++ Flags:   -std=c++98 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3 @H5_ECXXFLAGS@
                   AM C++ Flags:  -DOLD_HEADER_FILENAME -DHDF_NO_NAMESPACE -DNO_STATIC_CAST
             Shared C++ Library: yes
             Static C++ Library: yes

                           Java: no

Features:
---------
                   Parallel HDF5: no
Parallel Filtered Dataset Writes: no
              Large Parallel I/O: no
              High-level library: yes
                Build HDF5 Tests: yes
                Build HDF5 Tools: yes
                    Threadsafety: no
             Default API mapping: v110
  With deprecated public symbols: yes
          I/O filters (external): deflate(zlib),szip(encoder)
                             MPE: no
                      Direct VFD: no
                      Mirror VFD: no
              (Read-Only) S3 VFD: no
            (Read-Only) HDFS VFD: no
  Packages w/ extra debug output: none
                     API tracing: no
            Using memory checker: no
 Memory allocation sanity checks: no
          Function stack tracing: no
                Use file locking: best-effort
       Strict file format checks: no
    Optimization instrumentation: no
SMillerDev commented 9 months ago

Could you report this upstream? I don't think brew's modifications have much of an impact on this.

cho-m commented 9 months ago

It may be a side effect of moving to CMake build system.

CMake was requested in #157078 though there may be some issues https://github.com/Homebrew/homebrew-core/issues/157078#issuecomment-1885944263

Please check with upstream on feature compatibility and if there is specific build recommendations.

BenWibking commented 9 months ago

Yes, the CMake build of HDF5 is completely broken. I have built it from source dozens of times over the years, and it has never worked. Homebrew should build it with autotools.

BenWibking commented 9 months ago

It may be a side effect of moving to CMake build system.

CMake was requested in #157078 though there may be some issues #157078 (comment)

Please check with upstream on feature compatibility and if there is specific build recommendations.

Indeed, this was caused by https://github.com/Homebrew/homebrew-core/pull/159165, which should be reverted.

SMillerDev commented 9 months ago

We'll consider reverting if we can't get fixes submitted upstream. The goal for Homebrew is to fix any bugs we encounter upstream, not revert and wait until they magically fix themselves.

Please help by reporting the issue upstream and, if at all possible, thinking of some patches for the problem.

BenWibking commented 9 months ago

We'll consider reverting if we can't get fixes submitted upstream. The goal for Homebrew is to fix any bugs we encounter upstream, not revert and wait until they magically fix themselves.

Please help by reporting the issue upstream and, if at all possible, thinking of some patches for the problem.

It has been broken for literally decades. There is almost no chance of upstream fixing this.

SMillerDev commented 9 months ago

We won't know for sure until we try. All it might take is a bug report. We got some help on the CMake PR 🤷

Bo98 commented 9 months ago

Seems like it's expected behaviour and they don't want people relying on h5cc -show:

Upstream don't agree CMake is broken as they're planning to retire autotools builds: https://www.hdfgroup.org/2022/11/can-we-remove-the-autotools/

BenWibking commented 9 months ago

Unfortunately, CMake itself still relies on the use of h5cc -show to detect the compiler flags when linking against HDF5: https://github.com/Kitware/CMake/blob/e315d7cb19f019b91e68f7d3ca5720f294c70bab/Modules/FindHDF5.cmake#L12

So keeping this as-is breaks all builds of projects that use CMake's built-in functionality to link against HDF5. That is why all Linux distributions (AFAIK) build HDF5 via autotools.

Bo98 commented 9 months ago

Unfortunately, CMake itself still relies on the use of h5cc -show to detect the compiler flags when linking against HDF5: https://github.com/Kitware/CMake/blob/e315d7cb19f019b91e68f7d3ca5720f294c70bab/Modules/FindHDF5.cmake#L12

That code only runs if it can't find the HDF5 CMake config which should be available: https://github.com/Kitware/CMake/blob/e315d7cb19f019b91e68f7d3ca5720f294c70bab/Modules/FindHDF5.cmake#L509-L592. Unless you're disabling that with HDF5_NO_FIND_PACKAGE_CONFIG_FILE.

If that config is indeed missing then that is definitely a significant bug.

BenWibking commented 9 months ago

Unfortunately, CMake itself still relies on the use of h5cc -show to detect the compiler flags when linking against HDF5: https://github.com/Kitware/CMake/blob/e315d7cb19f019b91e68f7d3ca5720f294c70bab/Modules/FindHDF5.cmake#L12

That code only runs if it can't find the HDF5 CMake config which should be available: https://github.com/Kitware/CMake/blob/e315d7cb19f019b91e68f7d3ca5720f294c70bab/Modules/FindHDF5.cmake#L509-L592. Unless you're disabling that with HDF5_NO_FIND_PACKAGE_CONFIG_FILE.

If that config is indeed missing then that is definitely a significant bug.

It's not available. (We are not setting that CMake variable.) This problem caused all of our macOS CI builds to fail on GitHub actions runners.

Bo98 commented 9 months ago

Thanks for the info - that's actually a bug on our side which I'll fix.

eli-schwartz commented 9 months ago

Seems like it's expected behaviour and they don't want people relying on h5cc -show:

Unfortunately, CMake itself still relies on the use of h5cc -show to detect the compiler flags when linking against HDF5: https://github.com/Kitware/CMake/blob/e315d7cb19f019b91e68f7d3ca5720f294c70bab/Modules/FindHDF5.cmake#L12

So keeping this as-is breaks all builds of projects that use CMake's built-in functionality to link against HDF5. That is why all Linux distributions (AFAIK) build HDF5 via autotools.

This is also a chicken-and-egg problem as I discussed at https://github.com/Homebrew/homebrew-core/pull/159165#issuecomment-1887647158

The tl;dr here is that for autotools builds of hdf5, the only way to detect hdf5 is with h5cc -show. For third-party projects which build with cmake, the builtin cmake module requires h5cc -show and using the one distributed with hdf5 may or may not reliably work.

For third-party projects which build with literally any other build system on the planet, the cmake config files are totally useless and don't exist. So the only way to reliably detect hdf5 is to run h5cc -show manually.

Third-party projects which don't use cmake, cannot stop using h5cc -show, because linux distros etc. all ship autotools builds of hdf5, where only h5cc is available. They cannot start using pkg-config, because pkg-config is only provided by cmake, which no one packages.

Users are stuck in a holding pattern. And hdf5 claims to want to provide pkg-config files in their autotools builds, but has major commitment issues (this is a general theme of hdf5 when it comes to build systems, sadly).

As I commented on the other ticket:

And at minimum, the missing -lhdf5 is because of the well known problem that cmake's version of h5cc heard about the idea of "compatibility" and said "what's that, some kind of boring Linux thing?" and implemented the absolute minimum scaffolding needed to be invoked internally by their cmake config file, and absolutely nothing for existing users.

If for no other reason, distributors have to ship an autotools build because they need to support existing software that relies on h5cc working. That software cannot use pkg-config because distributors ship an autotools build. This recursive logic will be fixed exactly no sooner than whenever upstream hdf5 accepts that the request to ship a pkg-config file for autotools isn't a request that should be ignored and closed every time someone offers one, then left to rot forever

So, erm.

We'll consider reverting if we can't get fixes submitted upstream. The goal for Homebrew is to fix any bugs we encounter upstream, not revert and wait until they magically fix themselves.

Please help by reporting the issue upstream and, if at all possible, thinking of some patches for the problem.

Congratulations, you've officially discovered the problem that all the other distros already knew about and were unable to get fixed.

Experiment concluded, homebrew has successfully done its part in the FOSS ecosystem by discovering bugs in order to report them and think of patches for the problem.

My advice: now that you've discovered this, you've done your part and can revert it now.

In parallel, you can go try to communicate with hdf5 upstream to fix said bugs.

Things which are needed:

Allow projects to migrate from h5cc to pkg-config, on their own schedule that is NOT tied to "which build system was hdf5 built with".

If hdf5 cannot make this work, then it seems unreasonable to break existing projects in the wild over it?

eli-schwartz commented 9 months ago

The ticket has been incorrectly closed. The fact that h5cc -show doesn't work is a thing that doesn't work, regardless of what cmake does.

Whether or not cmake config files are available does not ameliorate the problem for users of build systems other than cmake, which currently run h5cc -show in e.g. configure.ac scripts.

Bo98 commented 9 months ago

They cannot start using pkg-config, because pkg-config is only provided by cmake, which no one packages.

This isn't entirely true given some Linux distros do package a pkg-config file.

I've checked and some distros ship a pkg-config file:

+ probably some others I've missed. A couple of the above do use CMake builds to achieve this.

Homebrew previously didn't because it stuck with the autotools build while some others used a blend of Autotools+CMake or potentially custom patched the gaps.

We've received bug reports because we didn't use the modern pkg-config/CMake system. There is no superior path like you claim. For example, some things like the h5py bindings only support pkg-config and do not support h5cc -show. It seems like it's a choice between supporting which half of the development community. A significant number of our dependents seem to support the newer system - meson may be the most notable exception though I'll keep checking for more.

Bo98 commented 9 months ago

The ticket has been incorrectly closed

The issue the original reporter described is now fixed, which was incompatibility with the CMake's FindHDF5 system due to an error on our part.

Bo98 commented 9 months ago

Actually meson seems work fine - it already supports pkg-config:

project('hdf5test', 'c')

h5c = dependency('hdf5', language: 'c')
The Meson build system
Version: 1.3.1
Source dir: /private/tmp
Build dir: /private/tmp/build
Build type: native build
Project name: hdf5test
Project version: undefined
C compiler for the host machine: cc (clang 15.0.0 "Apple clang version 15.0.0 (clang-1500.1.0.2.5)")
C linker for the host machine: cc ld64 1022.1
Host machine cpu family: aarch64
Host machine cpu: aarch64
Found pkg-config: YES (/opt/homebrew/bin/pkg-config) 0.29.2
Run-time dependency HDF5 for c found: YES 1.14.3
Build targets in project: 0

So that brings the known breakages back down to zero again, but if there's anything I've missed here then I'm happy to help out. I agree with you in that upstream really should have handled this better.

blawrence-ont commented 9 months ago

brew@1.14.3 still appears to be broken. It now gets past the configuration step that is was failing on previously, however when linking we see missing symbols:

ld: Undefined symbols:
  _SZ_BufftoBuffCompress, referenced from:
      _H5Z__filter_szip in libhdf5.a[341](H5Zszip.c.o)
  _SZ_BufftoBuffDecompress, referenced from:
      _H5Z__filter_szip in libhdf5.a[341](H5Zszip.c.o)
  _SZ_encoder_enabled, referenced from:
      _H5Z_init in libhdf5.a[335](H5Z.c.o)
  _compress2, referenced from:
      _H5Z__filter_deflate in libhdf5.a[336](H5Zdeflate.c.o)
  _inflate, referenced from:
      _H5Z__filter_deflate in libhdf5.a[336](H5Zdeflate.c.o)
  _inflateEnd, referenced from:
      _H5Z__filter_deflate in libhdf5.a[336](H5Zdeflate.c.o)
      _H5Z__filter_deflate in libhdf5.a[336](H5Zdeflate.c.o)
      _H5Z__filter_deflate in libhdf5.a[336](H5Zdeflate.c.o)
  _inflateInit_, referenced from:
      _H5Z__filter_deflate in libhdf5.a[336](H5Zdeflate.c.o)

Minimal repro case:

project(test)
set(HDF5_USE_STATIC_LIBRARIES ON)
find_package(HDF5 REQUIRED)
add_executable(test test.c)
target_link_libraries(test HDF5::HDF5)
// test.c
#include <hdf5.h>
int main() {
    H5Fopen("", 0, 0);
}

Enabling HDF5_FIND_DEBUG shows a difference in the libraries found:

Using hdf5@1.10:
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /opt/homebrew/Cellar/hdf5@1.10/1.10.11/lib/libhdf5.a;/opt/homebrew/opt/libaec/lib/libsz.dylib;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libz.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libdl.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libm.tbd (found version "1.10.11")  
-- HDF5_DIR: HDF5_DIR-NOTFOUND
-- HDF5_DEFINITIONS: 
-- HDF5_INCLUDE_DIRS: /opt/homebrew/Cellar/hdf5@1.10/1.10.11/include;/opt/homebrew/opt/libaec/include
-- HDF5_LIBRARIES: /opt/homebrew/Cellar/hdf5@1.10/1.10.11/lib/libhdf5.a;/opt/homebrew/opt/libaec/lib/libsz.dylib;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libz.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libdl.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libm.tbd
-- HDF5_HL_LIBRARIES: 
-- HDF5_C_DEFINITIONS: 
-- HDF5_C_INCLUDE_DIR: 
-- HDF5_C_INCLUDE_DIRS: /opt/homebrew/Cellar/hdf5@1.10/1.10.11/include;/opt/homebrew/opt/libaec/include
-- HDF5_C_LIBRARY: 
-- HDF5_C_LIBRARIES: /opt/homebrew/Cellar/hdf5@1.10/1.10.11/lib/libhdf5.a;/opt/homebrew/opt/libaec/lib/libsz.dylib;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libz.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libdl.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libm.tbd
-- HDF5_C_HL_LIBRARY: 
-- HDF5_C_HL_LIBRARIES: 
-- Defined targets (if any):
-- ... hdf5::hdf5

Using hdf5:
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Found HDF5: /opt/homebrew/lib/libhdf5.a (found version "1.14.3")  
-- HDF5_DIR: HDF5_DIR-NOTFOUND
-- HDF5_DEFINITIONS: 
-- HDF5_INCLUDE_DIRS: /opt/homebrew/include
-- HDF5_LIBRARIES: /opt/homebrew/lib/libhdf5.a
-- HDF5_HL_LIBRARIES: 
-- HDF5_C_DEFINITIONS: 
-- HDF5_C_INCLUDE_DIR: /opt/homebrew/include
-- HDF5_C_INCLUDE_DIRS: /opt/homebrew/include
-- HDF5_C_LIBRARY: 
-- HDF5_C_LIBRARIES: /opt/homebrew/lib/libhdf5.a
-- HDF5_C_HL_LIBRARY: 
-- HDF5_C_HL_LIBRARIES: 
-- Defined targets (if any):
-- ... hdf5::hdf5

Also note that if I have pkg-config installed then I get an error during configuration with hdf5 but not with hdf5@1.10:

CMake Error at /opt/homebrew/Cellar/cmake/3.28.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find HDF5 (missing: HDF5_LIBRARIES) (found version "1.14.3")
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.28.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /opt/homebrew/Cellar/cmake/3.28.1/share/cmake/Modules/FindHDF5.cmake:1025 (find_package_handle_standard_args)
  CMakeLists.txt:4 (find_package)
Bo98 commented 9 months ago

Thank you for the repro! Looks like there was a change on our side I forgot to push - try brew reinstall hdf5 now and see if it works again.

Tested your example now and it seems to work with the latest build.

blawrence-ont commented 9 months ago

Yep that's done the trick. Thanks for the fix!

eli-schwartz commented 9 months ago

This isn't entirely true given some Linux distros do package a pkg-config file.

Sure. Debian builds with autotools, and also installs a set of pkg-config files they wrote from scratch. They often do that for projects that don't provide pkg-config files at all, which is not particularly collaborative and people cannot assume they exist...

Alpine and Arch build and install with autotools, and also run cmake to generate pkg-config files only. I mentioned this somewhere I think.

My opinion on Nix can be best stated as "no comment", but I was indeed unaware that FreeBSD and spack used cmake to build. :/

Actually meson seems work fine - it already supports pkg-config:

Meson does support both h5cc and pkg-config. Meson uses dependency provider "methods" which are selectable, so using dependency('hdf5', method: 'config-tool') is broken. It is a "simple" fix, tell people to use "auto" or "pkg-config". Users may have in-house ecosystems where they need to enforce one or the other, but then there is nothing we can do about it and they'll have to make sure that their systems are configured the way they need it. :shrug: I'm actually more worried about autotools projects here, as fixing it is considerably less trivial.

For meson, I had proposed a meson patch on the 17th, which I'm about to land, that causes config-tool detection to abort with a "notfound" error when it detects that h5cc was produced via cmake. Since meson tests each dependency provider in CI to check that it works, we need both this and to annotate the test as "expected-fail" on macOS, since it... fails with homebrew hdf5.

I still think the correct solution here is to install a functional h5cc...

johnstairs commented 9 months ago

This seems to be broken on amd64 macos monterey, version 1.14.3 rebuild 3.

Sample logs from a GitHub actions run: https://github.com/microsoft/yardl/actions/runs/7633454119/job/20795750850 

CMake Error at /usr/local/Cellar/cmake/3.28.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find HDF5 (missing: HDF5_LIBRARIES) (found suitable version
-- Configuring incomplete, errors occurred!
  "1.14.3", minimum required is "1.10.5")

The "original" rebuild 0 worked.

theAeon commented 8 months ago

Can confirm its broken, but in an entirely different way for me.

amd64 on monterey works fine, according to this bit from the build artifacts of https://github.com/welch-lab/RcppPlanc/actions/runs/7919769437/job/21621294614#logs

-- Found HDF5: /usr/local/Cellar/hdf5/1.14.3/lib/libhdf5.dylib;/usr/local/opt/libaec/lib/libsz.dylib;/Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk/usr/lib/libz.tbd;/Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk/usr/lib/libdl.tbd;/Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk/usr/lib/libm.tbd (found version "1.14.3")

However, it DOESN'T work on arm64 sonoma (https://github.com/welch-lab/RcppPlanc/actions/runs/7919769437/job/21621294872)

-- Unable to determine HDF5 C flags from HDF5 wrapper.
CMake Error at /opt/homebrew/Cellar/cmake/3.28.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find HDF5 (missing: HDF5_LIBRARIES) (found version "1.14.3")
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.28.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /opt/homebrew/Cellar/cmake/3.28.1/share/cmake/Modules/FindHDF5.cmake:1025 (find_package_handle_standard_args)
  /private/var/folders/1k/qq3pcbf12vb6vyblh81736p40000gn/T/RtmphPB8Sj/Rbuild1a9321e32219/RcppPlanc/src/_deps/highfive-src/CMake/HighFiveTargetDeps.cmake:30 (find_package)
  /private/var/folders/1k/qq3pcbf12vb6vyblh81736p40000gn/T/RtmphPB8Sj/Rbuild1a9321e32219/RcppPlanc/src/_deps/highfive-src/CMakeLists.txt:98 (include)
-- Configuring incomplete, errors occurred!
ERROR: configuration failed for package ‘RcppPlanc’