Potential "awk" bug

[akhst7]

Hi,

I tried to update BPCells to 0.2 and run into a few compilation error and a following is the first error;

+ echo 'Failure running either '\''h5cc -show'\'' or: '\''h5cc -showconfig | awk -F: '\''/FLAGS|Extra libraries:/ {printf(%s' ', $2)}'\'''
Failure running either 'h5cc -show' or: 'h5cc -showconfig | awk -F: '/FLAGS|Extra libraries:/ {printf(%s , $2)}'

This awk one liner, h5cc -showconfig | awk -F: '/FLAGS|Extra libraries:/ {printf(%s , $2)} gives an following error;

awk: syntax error at source line 1
 context is
     >>> {printf(% <<< s, $2)}
awk: illegal statement at source line 1

I think this should be h5cc -showconfig | awk -F: '/FLAGS|Extra libraries:/ {printf("%s/n" , $2)}.

My system is as follows;

                      'c.          
                 ,xNMM.          -----------------------------
               .OMMMMo           OS: macOS 15.1 24B83 arm64
               OMMM0,            Host: Mac13,2
     .;loddo:' loolloddol;.      Kernel: 24.1.0
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Uptime: 13 hours, 26 mins
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    Packages: 245 (brew)
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: zsh 5.9
;MMMMMMMMMMMMMMMMMMMMMMMM:       Resolution: 3440x1440
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Quartz Compositor
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    WM Theme: Blue (Dark)
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal: iTerm2
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   Terminal Font: Monaco 10
    kMMMMMMMMMMMMMMMMMMMMMMd     CPU: Apple M1 Ultra
     ;KMMMMMMMWXXWMMMMMMMk.      GPU: Apple M1 Ultra
       .cooc,.    .,coo:.        Memory: 3984MiB / 131072MiB

[bnprks]

Hi @akhst7, thanks for the question. This error message is coming while BPCells is trying several different options to find the required HDF5 library. Failures in this part are expected, and will lead BPCells to re-try finding HDF5 from another source. For the awk one-liner, it looks like there is a typo in the error message, but not in the underlying code in the configure script (code link)

My guess for what you need to do is just run brew install hdf5 prior to installation to make sure you have the hdf5 dependency.

If that doesn't work, here are the steps I'd recommend quoted from the README (though you can just follow up with the error logs in this issue - no need to file a new one)

For a more verbose set of information, run Sys.setenv(BPCELLS_DEBUG_INSTALL="true") prior to remotes::install_github("bnprks/BPCells/r"). If you still can't solve the issue with that additional information, feel free to file a Github issue, being sure to use a collapsible section for the verbose installation log.

[akhst7]

@bnprks,
hdf5 is already installed by brew (see below). Looking at the configure script, if this awk line fails, these two variables won't be filled, HDF5_CFLAGS="" HDF5_LIBS="". I guess I can manually supply these in the configure script or my env and see what.

Also, are you planning to provide a BPCells v0.2 binary for M series Mac anytime soon ?

[akhst7]

Sorry I forgot to post the config output. It is as follows;

``` remotes::install_github("bnprks/BPCells/r") Using github PAT from envvar GITHUB_PAT. Use `gitcreds::gitcreds_set()` and unset GITHUB_PAT in .Renviron (or elsewhere) if you want to use the more secure git credential store instead. Downloading GitHub repo bnprks/BPCells@HEAD ── R CMD build ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ✔ checking for file ‘/private/var/folders/qp/qm7rgfps34v9p46d6mr3_80h0000gn/T/RtmpVGM6cO/remotes3b477d1ae039/bnprks-BPCells-1cdc199/r/DESCRIPTION’ ... ─ preparing ‘BPCells’: ✔ checking DESCRIPTION meta-information ... ─ cleaning src ─ running ‘cleanup’ ─ checking for LF line-endings in source and make files and shell scripts (373ms) ─ checking for empty or unneeded directories ─ building ‘BPCells_0.2.0.tar.gz’ Installing package into ‘/Volumes/Document/4.3.X_Rlib’ (as ‘lib’ is unspecified) * installing *source* package ‘BPCells’ ... ** using staged installation + '[' -z '' ']' + ENABLE_INSTALL_COUNTING=yes + '[' yes == yes ']' + curl --silent https://plausible.benparks.net/flask-plausible/bpcells-configure Recording install count metrics + echo 'Recording install count metrics' ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CC + CC=/opt/homebrew/opt/llvm/bin/clang ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXX + CXX=/opt/homebrew/opt/llvm/bin/clang++ + ENV_CFLAGS='-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include' + ENV_LDFLAGS='-L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/llvm/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/libomp/lib -L/opt/homebrew/opt/game-porting-toolkit-compiler/lib -L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib -lunwind' ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CFLAGS + CFLAGS='-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include' ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXXFLAGS + CXXFLAGS='-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include' ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config LDFLAGS + LDFLAGS='-L/opt/homebrew/opt/hdf5/lib -L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/libxml2/lib -L/opt/homebrew/opt/gettext/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib --sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -lomp -L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/llvm/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/libomp/lib -L/opt/homebrew/opt/game-porting-toolkit-compiler/lib -L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib -lunwind' + echo 'Testing hdf5 by compiling example program...' + HDF5_CFLAGS= + HDF5_LIBS=-lhdf5 + HDF5_OK= Testing hdf5 by compiling example program... + /opt/homebrew/opt/llvm/bin/clang tools/h5write.c -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include -L/opt/homebrew/opt/hdf5/lib -L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/libxml2/lib -L/opt/homebrew/opt/gettext/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib --sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -lomp -L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/llvm/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/libomp/lib -L/opt/homebrew/opt/game-porting-toolkit-compiler/lib -L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib -lunwind -lhdf5 -o tools/h5write tools/h5write.c:92:2: warning: no newline at end of file [-Wnewline-eof] 92 | } | ^ tools/h5write.c:31:12: warning: variable 'status' set but not used [-Wunused-but-set-variable] 31 | herr_t status; | ^ 2 warnings generated. ld: library 'unwind' not found clang: error: linker command failed with exit code 1 (use -v to see invocation) + '[' -z ']' + printf '\n\nRetrying with pkg-config flags...\n' Retrying with pkg-config flags... ++ pkg-config hdf5 --cflags + HDF5_CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.4.3/include ++ pkg-config hdf5 --libs + HDF5_LIBS='-L/opt/homebrew/Cellar/hdf5/1.14.4.3/lib -lhdf5' + /opt/homebrew/opt/llvm/bin/clang tools/h5write.c -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include -L/opt/homebrew/opt/hdf5/lib -L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/libxml2/lib -L/opt/homebrew/opt/gettext/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib --sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -lomp -L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/llvm/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/libomp/lib -L/opt/homebrew/opt/game-porting-toolkit-compiler/lib -L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib -lunwind -I/opt/homebrew/Cellar/hdf5/1.14.4.3/include -L/opt/homebrew/Cellar/hdf5/1.14.4.3/lib -lhdf5 -o tools/h5write tools/h5write.c:92:2: warning: no newline at end of file [-Wnewline-eof] 92 | } | ^ tools/h5write.c:31:12: warning: variable 'status' set but not used [-Wunused-but-set-variable] 31 | herr_t status; | ^ 2 warnings generated. ld: library 'unwind' not found clang: error: linker command failed with exit code 1 (use -v to see invocation) Searching for config information with h5cc -showconfig and h5cc -show... + '[' -z ']' + printf '\n\nSearching for config information with h5cc -showconfig and h5cc -show...\n' ++ h5cc -showconfig + H5CC_CONFIG='/opt/homebrew/h5cc dir is /opt/homebrew ++ awk -F: '/FLAGS|Extra libraries:/ {printf("%s ", $2)}' + H5CC_CONFIG=' m;dl -std=c99 -Wall;-Warray-bounds;-Wcast-qual;-Wconversion;-Wdouble-promotion;-Wextra;-Wformat=2;-Wframe-larger-than=16384;-Wimplicit-fallthrough;-Wnull-dereference;-Wunused-const-variable;-Wwrite-strings;-Wpedantic;-Wvolatile-register-var;-Wno-c++-compat;-Wbad-function-cast;-Wimplicit-function-declaration;-Wincompatible-pointer-types;-Wmissing-declarations;-Wpacked;-Wshadow;-Wswitch;-Wno-error=incompatible-pointer-types-discards-qualifiers;-Wunused-function;-Wunused-variable;-Wunused-parameter;-Wcast-align;-Wformat;-Wno-missing-noreturn ' ++ h5cc -show clang: error: unknown argument: '-show' + H5CC_CONFIG2='/opt/homebrew/h5cc dir is /opt/homebrew' + echo 'Failure running either '\''h5cc -show'\'' or: '\''h5cc -showconfig | awk -F: '\''/FLAGS|Extra libraries:/ {printf(%s' ', $2)}'\''' Failure running either 'h5cc -show' or: 'h5cc -showconfig | awk -F: '/FLAGS|Extra libraries:/ {printf(%s , $2)}' + '[' -z ']' + printf '\n\nSearching for hdf5 in a conda env...\n' Searching for hdf5 in a conda env... + '[' -z /opt/miniconda3 ']' + HDF5_CFLAGS=-I/opt/miniconda3/include + HDF5_LIBS='-Wl,-rpath,/opt/miniconda3/lib -L/opt/miniconda3/lib -lhdf5' + /opt/homebrew/opt/llvm/bin/clang tools/h5write.c -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -I/opt/homebrew/opt/hdf5/include -L/opt/homebrew/opt/hdf5/lib -L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/libxml2/lib -L/opt/homebrew/opt/gettext/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib --sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -lomp -L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/llvm/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/libomp/lib -L/opt/homebrew/opt/game-porting-toolkit-compiler/lib -L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib -lunwind -I/opt/miniconda3/include -Wl,-rpath,/opt/miniconda3/lib -L/opt/miniconda3/lib -lhdf5 -o tools/h5write tools/h5write.c:92:2: warning: no newline at end of file [-Wnewline-eof] 92 | } | ^ tools/h5write.c:31:12: warning: variable 'status' set but not used [-Wunused-but-set-variable] 31 | herr_t status; | ^ 2 warnings generated. ld: library 'unwind' not found clang: error: linker command failed with exit code 1 (use -v to see invocation) Unable to locate libhdf5. Please install manually or edit compiler flags. + '[' -z ']' + printf '\n\nUnable to locate libhdf5. Please install manually or edit compiler flags.\n' + exit 1 ``` SUMMARY OF THE HDF5 CONFIGURATION ================================= General Information: ------------------- HDF5 Version: 1.14.4.3 Configured on: 2024-05-22 Configured by: Unix Makefiles Host system: Darwin-24.0.0 Uname information: Darwin Byte sex: little-endian Installation point: /opt/homebrew/Cellar/hdf5/1.14.4.3 Compiling Options: ------------------ Build Mode: Release Debugging Symbols: OFF Asserts: OFF Profiling: OFF Optimization Level: OFF Linking Options: ---------------- Libraries: Statically Linked Executables: OFF LDFLAGS: H5_LDFLAGS: AM_LDFLAGS: Extra libraries: m;dl Archiver: /usr/bin/ar AR_FLAGS: Ranlib: /usr/bin/ranlib Languages: ---------- C: YES C Compiler: clang 16.0.0.16000026 CPPFLAGS: H5_CPPFLAGS: AM_CPPFLAGS: CFLAGS: -std=c99 H5_CFLAGS: -Wall;-Warray-bounds;-Wcast-qual;-Wconversion;-Wdouble-promotion;-Wextra;-Wformat=2;-Wframe-larger-than=16384;-Wimplicit-fallthrough;-Wnull-dereference;-Wunused-const-variable;-Wwrite-strings;-Wpedantic;-Wvolatile-register-var;-Wno-c++-compat;-Wbad-function-cast;-Wimplicit-function-declaration;-Wincompatible-pointer-types;-Wmissing-declarations;-Wpacked;-Wshadow;-Wswitch;-Wno-error=incompatible-pointer-types-discards-qualifiers;-Wunused-function;-Wunused-variable;-Wunused-parameter;-Wcast-align;-Wformat;-Wno-missing-noreturn AM_CFLAGS: Shared C Library: YES Static C Library: YES Fortran: ON Fortran Compiler: /opt/homebrew/bin/gfortran 14.2.0 Fortran Flags: H5 Fortran Flags: AM Fortran Flags: Shared Fortran Library: YES Static Fortran Library: YES Module Directory: /tmp/hdf5-20240926-8121-wnh6z0/hdf5-1.14.4-3/build/mod C++: ON C++ Compiler: clang++ 16.0.0.16000026 C++ Flags: H5 C++ Flags: -Wall;-Warray-bounds;-Wcast-qual;-Wconversion;-Wdouble-promotion;-Wextra;-Wformat=2;-Wframe-larger-than=16384;-Wimplicit-fallthrough;-Wnull-dereference;-Wunused-const-variable;-Wwrite-strings;-Wpedantic;-Wvolatile-register-var;-Wno-c++-compat;-Wno-missing-noreturn AM C++ Flags: Shared C++ Library: YES Static C++ Library: YES JAVA: OFF JAVA Compiler: Features: --------- Parallel HDF5: OFF Parallel Filtered Dataset Writes: Large Parallel I/O: High-level library: ON Dimension scales w/ new references: Build HDF5 Tests: OFF Build HDF5 Tools: ON Build GIF Tools: OFF Threadsafety: OFF Default API mapping: v114 With deprecated public symbols: ON I/O filters (external): DEFLATE DECODE ENCODE _Float16 support: OFF Map (H5M) API: OFF Direct VFD: OFF Mirror VFD: OFF Subfiling VFD: OFF (Read-Only) S3 VFD: OFF (Read-Only) HDFS VFD: OFF Packages w/ extra debug output: API Tracing: OFF Using memory checker: OFF Function Stack Tracing: OFF Use file locking: best-effort Strict File Format Checks: OFF Optimization Instrumentation: ' ++ echo '/opt/homebrew/h5cc dir is /opt/homebrew SUMMARY OF THE HDF5 CONFIGURATION ================================= General Information: ------------------- HDF5 Version: 1.14.4.3 Configured on: 2024-05-22 Configured by: Unix Makefiles Host system: Darwin-24.0.0 Uname information: Darwin Byte sex: little-endian Installation point: /opt/homebrew/Cellar/hdf5/1.14.4.3 Compiling Options: ------------------ Build Mode: Release Debugging Symbols: OFF Asserts: OFF Profiling: OFF Optimization Level: OFF Linking Options: ---------------- Libraries: Statically Linked Executables: OFF LDFLAGS: H5_LDFLAGS: AM_LDFLAGS: Extra libraries: m;dl Archiver: /usr/bin/ar AR_FLAGS: Ranlib: /usr/bin/ranlib Languages: ---------- C: YES C Compiler: clang 16.0.0.16000026 CPPFLAGS: H5_CPPFLAGS: AM_CPPFLAGS: CFLAGS: -std=c99 H5_CFLAGS: -Wall;-Warray-bounds;-Wcast-qual;-Wconversion;-Wdouble-promotion;-Wextra;-Wformat=2;-Wframe-larger-than=16384;-Wimplicit-fallthrough;-Wnull-dereference;-Wunused-const-variable;-Wwrite-strings;-Wpedantic;-Wvolatile-register-var;-Wno-c++-compat;-Wbad-function-cast;-Wimplicit-function-declaration;-Wincompatible-pointer-types;-Wmissing-declarations;-Wpacked;-Wshadow;-Wswitch;-Wno-error=incompatible-pointer-types-discards-qualifiers;-Wunused-function;-Wunused-variable;-Wunused-parameter;-Wcast-align;-Wformat;-Wno-missing-noreturn AM_CFLAGS: Shared C Library: YES Static C Library: YES Fortran: ON Fortran Compiler: /opt/homebrew/bin/gfortran 14.2.0 Fortran Flags: H5 Fortran Flags: AM Fortran Flags: Shared Fortran Library: YES Static Fortran Library: YES Module Directory: /tmp/hdf5-20240926-8121-wnh6z0/hdf5-1.14.4-3/build/mod C++: ON C++ Compiler: clang++ 16.0.0.16000026 C++ Flags: H5 C++ Flags: -Wall;-Warray-bounds;-Wcast-qual;-Wconversion;-Wdouble-promotion;-Wextra;-Wformat=2;-Wframe-larger-than=16384;-Wimplicit-fallthrough;-Wnull-dereference;-Wunused-const-variable;-Wwrite-strings;-Wpedantic;-Wvolatile-register-var;-Wno-c++-compat;-Wno-missing-noreturn AM C++ Flags: Shared C++ Library: YES Static C++ Library: YES JAVA: OFF JAVA Compiler: Features: --------- Parallel HDF5: OFF Parallel Filtered Dataset Writes: Large Parallel I/O: High-level library: ON Dimension scales w/ new references: Build HDF5 Tests: OFF Build HDF5 Tools: ON Build GIF Tools: OFF Threadsafety: OFF Default API mapping: v114 With deprecated public symbols: ON I/O filters (external): DEFLATE DECODE ENCODE _Float16 support: OFF Map (H5M) API: OFF Direct VFD: OFF Mirror VFD: OFF Subfiling VFD: OFF (Read-Only) S3 VFD: OFF (Read-Only) HDFS VFD: OFF Packages w/ extra debug output: API Tracing: OFF Using memory checker: OFF Function Stack Tracing: OFF Use file locking: best-effort Strict File Format Checks: OFF Optimization Instrumentation: '

Interestingly, the config picks Apple's clang not Brew's clang.

clang -v                                                                                                                                            [13:01:56]
Homebrew clang version 19.1.2
Target: arm64-apple-darwin24.1.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm/19.1.2/bin

[bnprks]

Hi @akhst7, thanks for the follow-up information!

For binary mac builds (and several other OS options), I've had BPCells hosted for a while on r-universe, though I see that hasn't made it properly to be listed in the README file. At any rate, it should be a one-liner install: install.packages('BPCells', repos = c('https://bnprks.r-universe.dev', 'https://cloud.r-project.org'))

As for the error that you're getting, it looks like BPCells is picking up some bad linker flags from your LDFLAGS environment variable (The line starting with + ENV_LDFLAGS='-L/opt/homebrew/opt/llvm/lib). This includes -lunwind which is causing all of the test compilations to fail with ld: library 'unwind' not found.

If you want to build BPCells from source, you'll need to modify that environment variable so it doesn't cause compilations to fail. The easiest way would probably be to clear it within the R session itself, e.g. via Sys.unsetenv("LDFLAGS") prior to the remotes::install_github("bnprks/BPCells/r") call.

Hope that helps!

-Ben

P.S. The HDF5_CFLAGS and HDF5_LIBS are still being handled correctly as best as I can tell. Throughout the configure script they get filled with several different attempts to find flags that will allow a test hdf5 program to compile without errors. Though in this case your LDFLAGS environment variable seems to be causing errors with all compilations regardless of the HDF5 setup.

[akhst7]

Hi Ben, I am still having the same issue but I will toy with a configure script to see what. I will try the binary but I wish the configure script would accept a custom Makervers rather than etc/Makevers.

Thanks.

Aki