search

bnprks / BPCells

Scaling Single Cell Analysis to Millions of Cells
https://bnprks.github.io/BPCells
Other
156 stars 17 forks source link

Can't install BPCells on MacOS #24

Closed Dario-Rocha closed 1 year ago

Dario-Rocha commented 1 year ago

The Error:

remotes::install_github("bnprks/BPCells")
Downloading GitHub repo bnprks/BPCells@HEAD
── R CMD build ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/private/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T/RtmpXZPBBi/remotes4497101206fb/bnprks-BPCells-c647fb1/DESCRIPTION’ ...
─  preparing ‘BPCells’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts (423ms)
─  checking for empty or unneeded directories
─  building ‘BPCells_0.1.0.tar.gz’

* installing *source* package ‘BPCells’ ...
** using staged installation
Testing hdf5 by compiling example program...

Retrying with pkg-config flags...
Error running 'pkg-config hdf5 --cflags --libs'

Searching for hdf5 in a conda env...
no conda environment found in '$CONDA_PREFIX'

Unable to locate libhdf5. Please install manually or edit compiler flags.
ERROR: configuration failed for package ‘BPCells’
* removing ‘/Library/Frameworks/R.framework/Versions/4.2/Resources/library/BPCells’
Warning message:
In i.p(...) :
  installation of package ‘/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T//RtmpXZPBBi/file449714bda8bf/BPCells_0.1.0.tar.gz’ had non-zero exit status

First I installed Macports and HDF5 with sudo port install hdf5 having the base MacOS environment activated (as default in the terminal) The error persisted

Then I installed HDF5 using homebrew having the base MacOS environment activated (as default in the terminal) following this post. The error persisted.

Then I installed HDF5 with conda having the base env activated % conda install -c anaconda hdf5 But the error persisted

I've tried, in R

Sys.setenv(HDF5_DIR = "/opt/homebrew/Cellar/hdf5/1.14.0")
Sys.setenv(HDF5_LIBRARIES = "/opt/homebrew/Cellar/hdf5/1.14.0/lib")
remotes::install_github("bnprks/BPCells")

To no avail

What else could I try? Thank you in advance for your time

bnprks commented 1 year ago

It looks like you have already identified the directories containing the hdf5 headers (hfd5.h is presumably in /opt/homebrew/Cellar/hdf5/1.14.0/include) and dynamic libraries (/opt/homebrew/Cellar/hdf5/1.14.0/lib). Therefore, I'd try putting the following in your ~/.R/Makevars file:

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib

Then with those changes and conda deactivated, try installing BPCells.

If that continues to not work, there's a newly added debugging tool to help get more error diagnostics from the installation process. On the terminal, run export BPCELLS_DEBUG_INSTALL=true on the command line before starting R on the command line and running remotes::install_github("bnprks/BPCells"). This will print out the commands the installer runs to check for the hdf5 libraries, as well as any error messages printed out during the test compilations.

bnprks commented 1 year ago

To help make these problems less common I just tried making a fix in 5d9e78c4e0a, which will also try to use h5cc -show and h5cc -showconfig to try to automatically infer the required compiler flags.

If h5cc -show prints something for you in the terminal, you might try just re-installing with no special configuration set to see if this fix works for you.

(p.s. if you still are ending up with issues, could you let me know your MacOS version and the compiler version you're using?)

Dario-Rocha commented 1 year ago

Thank you for your reply, I spotted some mistakes I made and tried again, though without sucess

When trying to manually modify .R/Makevars with a text editor I realized that the file simply didn't exist. So I: -Uninstalled hdf5 with homebrew -Deactivated the base conda environment -Installed hdf5 with homebrew -Confirmed hdf5 installation directory

~ % brew info llvm
==> llvm: stable 16.0.3 (bottled), HEAD [keg-only]
Next-gen compiler infrastructure
https://llvm.org/
/opt/homebrew/Cellar/llvm/16.0.3 (6,779 files, 1.5GB)

-Confirmed the location of clang and clang++

 ~ % ls /opt/homebrew/Cellar/llvm/16.0.3/bin/clang
/opt/homebrew/Cellar/llvm/16.0.3/bin/clang
~ % ls /opt/homebrew/Cellar/llvm/16.0.3/bin/clang++
/opt/homebrew/Cellar/llvm/16.0.3/bin/clang++

Created /Users/dariorocha/ .R/Makevars.rtf using a text editor

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib
CC=/opt/homebrew/Cellar/llvm/16.0.3/bin/clang
CXX=/opt/homebrew/Cellar/llvm/16.0.3/bin/clang++

-ran this on the terminal export BPCELLS_DEBUG_INSTALL=true

-then tried to install BPCells again

remotes::install_github("bnprks/BPCells")
Downloading GitHub repo bnprks/BPCells@HEAD
── R CMD build ───────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/private/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T/Rtmp6Qj2kr/remotes3f2510dca9b5/bnprks-BPCells-98536c9/DESCRIPTION’ ...
─  preparing ‘BPCells’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts (461ms)
─  checking for empty or unneeded directories
─  building ‘BPCells_0.1.0.tar.gz’

* installing *source* package ‘BPCells’ ...
** using staged installation
Testing hdf5 by compiling example program...

Retrying with pkg-config flags...
Error running 'pkg-config hdf5 --cflags --libs'

Searching for config information with h5cc -showconfig and h5cc -show...
Found working hdf5
HDF5_CFLAGS=' -I/Users/dariorocha/miniconda3/include'
HDF5_LIBS=' -Wl,-rpath,/Users/dariorocha/miniconda3/lib -L/Users/dariorocha/miniconda3/lib -lcrypto -lcurl -lpthread -lz -ldl -lm -Wl,-rpath,/Users/dariorocha/miniconda3/lib -lhdf5'

Testing architecture flag support support...'-march=native' succeeded
ARCH_FLAG='-march=native'

Unable to compile program with C++17 std::filesystem.
Please install a newer compiler version and set CC and CXX in ~/.R/Makevars

-moved on to check h5cc

~ % h5cc -show                                     
clang -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5_hl.a /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5.a -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm

clang version is

~ % clang --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
~ % clang++ --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

System info is:

% sw_vers
ProductName:        macOS
ProductVersion:     13.3.1
ProductVersionExtra:    (a)
BuildVersion:       22E772610a
bnprks commented 1 year ago

It looks like the debug log didn't quite come through properly. When you run export BPCELLS_DEBUG_INSTALL=true, you then need to run R and remotes::install_github() in the same terminal. It should print out a bunch of lines starting with "+", interspersed with some program output. That said, I have a few ideas from the information you've provided

  1. I see you're on an ARM Mac. In issue #14, it seems that just using the homebrew-provided hdf5 is likely sufficient, and you are unlikely to need the llvm from homebrew as well.
    • I would try removing the CC and CXX entries from your Makevars
  2. Based on the log you provided, it seems that your compilation used a conda hdf5 install, rather than homebrew as you intended. It's not entirely clear to me why this is happening. You might try removing the CC and CXX entries as I mentioned, or simply deactivating conda in your terminal prior to running R and remotes::install_github(). This will prevent the installation from picking up the conda version of hdf5 at least. Otherwise, double-check CFLAGS, CXXFLAGS, and LDFLAGS
  3. This error "Unable to compile program with C++17 std::filesystem." typically only shows up on older Intel Macs, so I'm not quite sure of the cause. I know mixups between ARM and x86 binaries can happen on ARM Macs as some programs still use Rosetta 2

Hopefully some of that is useful.

If not, please retry the export BPCELLS_DEBUG_INSTALL=true as I described, deactivate conda in the same terminal, then run R and provide the (much more verbose) debug log.

Dario-Rocha commented 1 year ago

Alright, another attempt:

-deleted the " .R/" directory and created it again without the initial blank space, then created the Makevars file and edited it, also adding the .txt extension, following this post: https://stackoverflow.com/questions/5891365/mac-os-x-doesnt-allow-to-name-files-starting-with-a-dot-how-do-i-name-the-hta/51788836#51788836

-My Makevars looks like this now

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib

-Opened the terminal, conda deactivate

-Checked pkg config

~ % pkg-config hdf5 --cflags --libs
Package hdf5 was not found in the pkg-config search path.
Perhaps you should add the directory containing `hdf5.pc'
to the PKG_CONFIG_PATH environment variable
No package 'hdf5' found

-Rechecked that hdf5 is installed with homebrew

~ % brew info hdf5
==> hdf5: stable 1.14.0 (bottled)
File format designed to store large amounts of data
https://www.hdfgroup.org/HDF5
Conflicts with:
  hdf5-mpi (because hdf5-mpi is a variant of hdf5, one can only use one or the other)
/opt/homebrew/Cellar/hdf5/1.14.0 (283 files, 20.6MB) *
  Poured from bottle using the formulae.brew.sh API on 2023-05-10 at 11:36:11
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/hdf5.rb
License: BSD-3-Clause
==> Dependencies
Build: autoconf ✘, automake ✘, libtool ✘
Required: gcc ✔, libaec ✔
==> Analytics
install: 613 (30 days), 12,491 (90 days), 391,867 (365 days)
install-on-request: 157 (30 days), 3,285 (90 days), 93,417 (365 days)
build-error: 0 (30 days)

-Checked that pkg config is installed

~ % brew info pkg-config
==> pkg-config: stable 0.29.2 (bottled)
Manage compile and link flags for libraries
https://freedesktop.org/wiki/Software/pkg-config/
/opt/homebrew/Cellar/pkg-config/0.29.2_3 (11 files, 676.2KB) *
  Poured from bottle using the formulae.brew.sh API on 2023-05-04 at 14:55:43
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/pkg-config.rb
License: GPL-2.0-or-later
==> Analytics
install: 4,778 (30 days), 81,050 (90 days), 2,104,367 (365 days)
install-on-request: 756 (30 days), 16,512 (90 days), 242,040 (365 days)
build-error: 17 (30 days)

-There is no pkgconfig directory in /opt/homebrew/Cellar/hdf5/1.14.0/lib

~ % ls /opt/homebrew/Cellar/hdf5/1.14.0/lib
libhdf5.310.dylib       libhdf5_hl.a
libhdf5.a           libhdf5_hl.dylib
libhdf5.dylib           libhdf5_hl_cpp.310.dylib
libhdf5_cpp.310.dylib       libhdf5_hl_cpp.a
libhdf5_cpp.a           libhdf5_hl_cpp.dylib
libhdf5_cpp.dylib       libhdf5_hl_fortran.a
libhdf5_fortran.310.dylib   libhdf5hl_fortran.310.dylib
libhdf5_fortran.a       libhdf5hl_fortran.a
libhdf5_fortran.dylib       libhdf5hl_fortran.dylib
libhdf5_hl.310.dylib

-Following chatbot instructions

sudo mkdir -p /opt/homebrew/lib/pkgconfig
sudo nano /opt/homebrew/lib/pkgconfig/hdf5.pc

-Then copied and pasted this in the file

prefix=/opt/homebrew/Cellar/hdf5/1.14.0
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: hdf5
Description: Hierarchical Data Format 5 (HDF5)
Version: 1.14.0

Libs: -L${libdir} -lhdf5
Cflags: -I${includedir}

-and save the file by pressing Ctrl+X, then Y, and finally Enter.

-now I can successfully run the command pkg-config hdf5 --cflags --libs

~ % pkg-config hdf5 --cflags --libs
-I/opt/homebrew/Cellar/hdf5/1.14.0/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5

So, within the terminal and with the base environment deactivated, I run R and try to install BPCells

Install log (EDIT by bnprks to make into collapsible section) ``` > remotes::install_github("bnprks/BPCells") Downloading GitHub repo bnprks/BPCells@HEAD ── R CMD build ───────────────────────────────────────────────────────────────── checking for file ‘/private/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T/R✔ checking for file ‘/private/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T/RtmphisDvG/remotes3c9d3c210373/bnprks-BPCells-32fb937/DESCRIPTION’ ─ preparing ‘BPCells’: ✔ checking DESCRIPTION meta-information ... ─ cleaning src ─ running ‘cleanup’ ─ checking for LF line-endings in source and make files and shell scripts (511ms) ─ checking for empty or unneeded directories ─ building ‘BPCells_0.1.0.tar.gz’ * installing *source* package ‘BPCells’ ... ** using staged installation + ENABLE_INSTALL_COUNTING=yes + '[' -n yes ']' + curl --silent https://plausible.benparks.net/flask-plausible/bpcells-configure + true ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CC + CC='clang -mmacosx-version-min=10.13' ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXX + CXX='clang++ -mmacosx-version-min=10.13 -std=gnu++14' ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CFLAGS + CFLAGS='-Wall -g -O2' ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXXFLAGS + CXXFLAGS='-Wall -g -O2' ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config LDFLAGS + LDFLAGS=-L/usr/local/lib + echo 'Testing hdf5 by compiling example program...' Testing hdf5 by compiling example program... + HDF5_CFLAGS= + HDF5_LIBS=-lhdf5 + HDF5_OK= + clang -mmacosx-version-min=10.13 tools/h5write.c -Wall -g -O2 -L/usr/local/lib -lhdf5 -o tools/h5write tools/h5write.c:19:10: fatal error: 'hdf5.h' file not found #include "hdf5.h" ^~~~~~~~ 1 error generated. + '[' -z ']' + printf '\n\nRetrying with pkg-config flags...\n' Retrying with pkg-config flags... ++ pkg-config hdf5 --cflags + HDF5_CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include ++ pkg-config hdf5 --libs + HDF5_LIBS='-L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5' + clang -mmacosx-version-min=10.13 tools/h5write.c -Wall -g -O2 -L/usr/local/lib -I/opt/homebrew/Cellar/hdf5/1.14.0/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5 -o tools/h5write tools/h5write.c:31:12: warning: variable 'status' set but not used [-Wunused-but-set-variable] herr_t status; ^ 1 warning generated. ld: warning: ignoring file /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64 Undefined symbols for architecture x86_64: "_H5Dclose", referenced from: _main in h5write-e7dbda.o "_H5Dcreate2", referenced from: _main in h5write-e7dbda.o "_H5Dwrite", referenced from: _main in h5write-e7dbda.o "_H5Fclose", referenced from: _main in h5write-e7dbda.o "_H5Fcreate", referenced from: _main in h5write-e7dbda.o "_H5Sclose", referenced from: _main in h5write-e7dbda.o "_H5Screate_simple", referenced from: _main in h5write-e7dbda.o "_H5T_NATIVE_INT_g", referenced from: _main in h5write-e7dbda.o "_H5Tclose", referenced from: _main in h5write-e7dbda.o "_H5Tcopy", referenced from: _main in h5write-e7dbda.o "_H5Tset_order", referenced from: _main in h5write-e7dbda.o "_H5check_version", referenced from: _main in h5write-e7dbda.o "_H5open", referenced from: _main in h5write-e7dbda.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) + '[' -z ']' + printf '\n\nSearching for config information with h5cc -showconfig and h5cc -show...\n' Searching for config information with h5cc -showconfig and h5cc -show... ++ h5cc -showconfig + H5CC_CONFIG=' SUMMARY OF THE HDF5 CONFIGURATION ================================= General Information: ------------------- HDF5 Version: 1.14.0 Configured on: Mon May 1 09:04:43 UTC 2023 Configured by: brew@Ventura-arm64.local Host system: aarch64-apple-darwin22.4.0 Uname information: Darwin Ventura-arm64.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:55:35 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_VMAPPLE arm64 Byte sex: little-endian Installation point: /opt/homebrew/Cellar/hdf5/1.14.0 Compiling Options: ------------------ Build Mode: production Debugging Symbols: no Asserts: no Profiling: no Optimization Level: high Linking Options: ---------------- Libraries: static, shared Statically Linked Executables: LDFLAGS: H5_LDFLAGS: -Wl,-commons,use_dylibs AM_LDFLAGS: -L/opt/homebrew/opt/libaec/lib Extra libraries: -lsz -lz -ldl -lm Archiver: ar AR_FLAGS: cr Ranlib: ranlib Languages: ---------- C: yes C Compiler: clang ( Apple clang version 14.0.3 ) CPPFLAGS: H5_CPPFLAGS: -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling AM_CPPFLAGS: -I/opt/homebrew/opt/libaec/include C Flags: H5 C Flags: -std=c99 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3 AM C Flags: Shared C Library: yes Static C Library: yes Fortran: yes Fortran Compiler: /opt/homebrew/opt/gcc/bin/gfortran ( GNU Fortran (Homebrew GCC 13.1.0) 13.1.0) Fortran Flags: H5 Fortran Flags: -std=f2008 -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow -pedantic -Warray-temporaries -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type -Winteger-division -Wfrontend-loop-interchange -fdiagnostics-urls=never -fno-diagnostics-color -s -Wno-unused-dummy-argument -O3 AM Fortran Flags: Shared Fortran Library: yes Static Fortran Library: yes Module Directory: ${includedir} C++: yes C++ Compiler: clang++ ( Apple clang version 14.0.3 ) C++ Flags: H5 C++ Flags: -std=c++11 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3 AM C++ Flags: Shared C++ Library: yes Static C++ Library: yes Java: no Features: --------- Parallel HDF5: no Parallel Filtered Dataset Writes: no Large Parallel I/O: no High-level library: yes Dimension scales w/ new references: no Build HDF5 Tests: yes Build HDF5 Tools: yes Build GIF Tools: no Threadsafety: no Default API mapping: v114 With deprecated public symbols: yes I/O filters (external): deflate(zlib),szip(encoder) Map (H5M) API: no Direct VFD: no Mirror VFD: no Subfiling VFD: no (Read-Only) S3 VFD: no (Read-Only) HDFS VFD: no Packages w/ extra debug output: none API tracing: no Using memory checker: no Function stack tracing: no Use file locking: best-effort Strict file format checks: no Optimization instrumentation: no' ++ echo ' SUMMARY OF THE HDF5 CONFIGURATION ================================= General Information: ------------------- HDF5 Version: 1.14.0 Configured on: Mon May 1 09:04:43 UTC 2023 Configured by: brew@Ventura-arm64.local Host system: aarch64-apple-darwin22.4.0 Uname information: Darwin Ventura-arm64.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:55:35 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_VMAPPLE arm64 Byte sex: little-endian Installation point: /opt/homebrew/Cellar/hdf5/1.14.0 Compiling Options: ------------------ Build Mode: production Debugging Symbols: no Asserts: no Profiling: no Optimization Level: high Linking Options: ---------------- Libraries: static, shared Statically Linked Executables: LDFLAGS: H5_LDFLAGS: -Wl,-commons,use_dylibs AM_LDFLAGS: -L/opt/homebrew/opt/libaec/lib Extra libraries: -lsz -lz -ldl -lm Archiver: ar AR_FLAGS: cr Ranlib: ranlib Languages: ---------- C: yes C Compiler: clang ( Apple clang version 14.0.3 ) CPPFLAGS: H5_CPPFLAGS: -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling AM_CPPFLAGS: -I/opt/homebrew/opt/libaec/include C Flags: H5 C Flags: -std=c99 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3 AM C Flags: Shared C Library: yes Static C Library: yes Fortran: yes Fortran Compiler: /opt/homebrew/opt/gcc/bin/gfortran ( GNU Fortran (Homebrew GCC 13.1.0) 13.1.0) Fortran Flags: H5 Fortran Flags: -std=f2008 -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow -pedantic -Warray-temporaries -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type -Winteger-division -Wfrontend-loop-interchange -fdiagnostics-urls=never -fno-diagnostics-color -s -Wno-unused-dummy-argument -O3 AM Fortran Flags: Shared Fortran Library: yes Static Fortran Library: yes Module Directory: ${includedir} C++: yes C++ Compiler: clang++ ( Apple clang version 14.0.3 ) C++ Flags: H5 C++ Flags: -std=c++11 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3 AM C++ Flags: Shared C++ Library: yes Static C++ Library: yes Java: no Features: --------- Parallel HDF5: no Parallel Filtered Dataset Writes: no Large Parallel I/O: no High-level library: yes Dimension scales w/ new references: no Build HDF5 Tests: yes Build HDF5 Tools: yes Build GIF Tools: no Threadsafety: no Default API mapping: v114 With deprecated public symbols: yes I/O filters (external): deflate(zlib),szip(encoder) Map (H5M) API: no Direct VFD: no Mirror VFD: no Subfiling VFD: no (Read-Only) S3 VFD: no (Read-Only) HDFS VFD: no Packages w/ extra debug output: none API tracing: no Using memory checker: no Function stack tracing: no Use file locking: best-effort Strict file format checks: no Optimization instrumentation: no' ++ awk -F: '/FLAGS|Extra libraries:/ {printf("%s ", $2)}' + H5CC_CONFIG=' -Wl,-commons,use_dylibs -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm cr -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include ' ++ h5cc -show + H5CC_CONFIG2='clang -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5_hl.a /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5.a -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm' + HDF5_CFLAGS= + HDF5_LIBS= + HDF5_RPATH_LAST_ARG=false + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr false = true + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr '' : '.*-L/opt/homebrew/opt/libaec/lib' + HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib' : '.*-lsz' + HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib -lsz' : '.*-lz' + HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz' : '.*-ldl' + HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl' : '.*-lm' + HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr '' : '.*-I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling' + HDF5_CFLAGS=' -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling' : '.*-I/opt/homebrew/opt/libaec/include' + HDF5_CFLAGS=' -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include' : '.*-I/opt/homebrew/opt/libaec/include' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm' : '.*-L/opt/homebrew/Cellar/hdf5/1.14.0/lib' + HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-L/opt/homebrew/opt/libaec/lib' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-lsz' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-lz' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-ldl' + for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2' + case "$arg" in + expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-lm' + HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5' + clang -mmacosx-version-min=10.13 tools/h5write.c -Wall -g -O2 -L/usr/local/lib -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5 -o tools/h5write tools/h5write.c:19:10: fatal error: 'hdf5.h' file not found #include "hdf5.h" ^~~~~~~~ 1 error generated. + '[' -z ']' + printf '\n\nSearching for hdf5 in a conda env...\n' Searching for hdf5 in a conda env... + '[' -z '' ']' + echo 'no conda environment found in '\''$CONDA_PREFIX'\''' no conda environment found in '$CONDA_PREFIX' + '[' -z ']' + printf '\n\nUnable to locate libhdf5. Please install manually or edit compiler flags.\n' Unable to locate libhdf5. Please install manually or edit compiler flags. + exit 1 ERROR: configuration failed for package ‘BPCells’ * removing ‘/Library/Frameworks/R.framework/Versions/4.2/Resources/library/BPCells’ ```
bnprks commented 1 year ago

Thanks for the debug log, that gave a lot of useful information.

It looks like your main problem is having a mixup between software for the x86_64 architecture, and software for the arm_64 architecture. x86_64 is emulated through Rosetta2 on ARM macs, while arm_64 is what the computer runs natively. Depending on whether your version of R is running as x86_64 or arm_64, you will need to match your compilers and libraries to that architecture. As I don't own one of these Macs personally, I don't have many useful tips on how to accomplish that, but it is definitely the source of the error in the log you just sent.

This issue has a bit of useful discussion about having mismatched x86_64/arm_64 when installing BPCells: #6. In that case, switching to running R natively in the arm_64 architecture helped solve the problem. If using x86_64 is important to you for some reason, then it appears my advice to use homebrew hdf5 over conda hdf5 is probably bad, though I've also noticed mac users having trouble with trying to install with hdf5 through conda

Some of my observations:

  1. In your error message, I notice this part in particular:

    + clang -mmacosx-version-min=10.13 tools/h5write.c -Wall -g -O2 -L/usr/local/lib -I/opt/homebrew/Cellar/hdf5/1.14.0/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5 -o tools/h5write
tools/h5write.c:31:12: warning: variable 'status' set but
     not used [-Wunused-but-set-variable]
   herr_t status;
          ^
1 warning generated.
ld: warning: ignoring file /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

    This bit "building for macOS-x86_64 but attempting to link with file built for macOS-arm64" is why you have the linker errors just below, and why I've diagnosed your main problem this way.

  2. Your makevars file must have exactly the name ~/.R/Makevars, and it cannot have a txt extension. These lines of the log indicate that R is not reading the CFLAGS, CXXFLAGS, and LDFLAGS you have set, likely due to the file having the wrong name:

    ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CFLAGS
+ CFLAGS='-Wall -g -O2'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXXFLAGS
+ CXXFLAGS='-Wall -g -O2'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config LDFLAGS
+ LDFLAGS=-L/usr/local/lib

  3. Don't worry about making pkg-config work -- that's mainly for use on linux. BPCells has several options for how to try to find hdf5, so it's okay if pkgconfig doesn't work. I would recommend you don't bother making a hdf5.pc file manually. If you set up ~/.R/Makevars correctly, you should succeed after the line Testing hdf5 by compiling example program..., and you shouldn't ever get to the line Retrying with pkg-config flags....

  4. Similarly, it looks like the h5cc -show option doesn't yield useful compiler flags on your system, so don't worry too much about it and just focus on getting the method with ~/.R/Makevars to work.

  5. For future error logs, you can use a collapsible section as described here to take up a bit less scroll space

Dario-Rocha commented 1 year ago

Sorry for the long logs, I was unaware of the collapsible section

Thank you for pointing out it was about the architecture. I decided to go from a clean slate, uninstalled R and Rstudio, then from the terminal deactivated the 'base' conda environment, installed R with homebrew, installed Rstudio with homebrew, created the .R directory mkdir -p ~/.R created the Makevars file touch ~/.R/Makevars edited the Makevars file from Rstudio file.edit("~/.Makevars")

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib

installed BPCells from Rstudio remotes::install_github("bnprks/BPCells") Installation was successful, haven't yet tested if everything works alright.

Thank you so much for your time

bnprks commented 1 year ago

Great, I'm glad you've gotten it installed finally! I'd suggest walking through this tutorial if you want to check everything is working.

rhodesch commented 8 months ago

Sorry for the long logs, I was unaware of the collapsible section

Thank you for pointing out it was about the architecture. I decided to go from a clean slate, uninstalled R and Rstudio, then from the terminal deactivated the 'base' conda environment, installed R with homebrew, installed Rstudio with homebrew, created the .R directory mkdir -p ~/.R created the Makevars file touch ~/.R/Makevars edited the Makevars file from Rstudio file.edit("~/.Makevars")

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib

installed BPCells from Rstudio remotes::install_github("bnprks/BPCells") Installation was successful, haven't yet tested if everything works alright.

Thank you so much for your time

Just confirming comment above works for me too (as would @bnprks comment on May 4, 2023) with: MacOS 14.3.1, M1 R version 4.3.2 hdf5 1.14.3 (via homebrew) no conda active during installation steps