installation challenges

[bbolker]

Nothing here yet; this is a placeholder for collecting installation issues into a single thread.

[mikeroswell]

Running MacOS 14.3 Sonoma and wasn't able to install epigrowthfit (or TMB from source).

Attached, please find the R console outputs from installation (with Rsession info at the top). Makevars and Makeconfig pasted below. Thanks for any tips!

Rsession_and_inst_messages.txt

mikeroswell@Michaels-MBP ~ % cat ~/.R/Makevars

LDFLAGS += -L/opt/homebrew/opt/libomp/lib
CPPFLAGS += -I/opt/homebrew/opt/libomp/include
FLIBS   =-L/opt/homebrew/opt/gfortran/lib
F77     = /opt/homebrew/bin/gfortran
FC      = /opt/homebrew/bin/gfortran

CFLAGS   = -I/opt/homebrew/include
CPPFLAGS = -I/opt/homebrew/include
CXXFLAGS = -I/opt/homebrew/include

SHLIB_OPENMP_CFLAGS=-Xpreprocessor -fopenmp
SHLIB_OPENMP_CXXFLAGS=-Xpreprocessor -fopenmp

mikeroswell@Michaels-MBP ~ % cat $(R RHOME)/etc/Makeconf
# etc/Makeconf.  Generated from Makeconf.in by configure.
#
# ${R_HOME}/etc/Makeconf
#
# R was configured using the following call
# (not including env. vars and site configuration)
# configure  'FW_VERSION=4.3-arm64' 'CC=clang -arch arm64' 'CXX=clang++ -arch arm64' 'OBJC=clang -arch arm64' 'FC=/opt/gfortran/bin/gfortran -arch arm64' 'CFLAGS=-falign-functions=64 -Wall -g -O2' 'CXXFLAGS=-falign-functions=64 -Wall -g -O2' 'OBJCFLAGS=-falign-functions=64 -Wall -g -O2' 'FCFLAGS=-Wall -g -O2' 'F77FLAGS=-Wall -g -O2' 'LDFLAGS=-L/opt/R/arm64/lib' 'CPPFLAGS=-I/opt/R/arm64/include' '--enable-memory-profiling' '--enable-R-framework' '--with-tcl-config=/opt/R/arm64/lib/tclConfig.sh' '--with-tk-config=/opt/R/arm64/lib/tkConfig.sh' '--build=aarch64-apple-darwin20' 'build_alias=aarch64-apple-darwin20' 'PKG_CONFIG_PATH=/usr/lib/pkgconfig:/opt/R/arm64/lib/pkgconfig:/opt/R/arm64/share/pkgconfig:/opt/X11/lib/pkgconfig:/opt/X11/share/pkgconfig'

## This fails if it contains spaces, or if it is quoted
include $(R_SHARE_DIR)/make/vars.mk

AR = ar
BLAS_LIBS = -L"$(R_HOME)/lib$(R_ARCH)" -lRblas

## C compilers
CC = clang -arch arm64
CFLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CC17 = clang -arch arm64 -std=gnu17
C17FLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CC23 = clang -arch arm64 -std=gnu2x
C23FLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CC90 = clang -arch arm64 -std=gnu90
C90FLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CC99 = clang -arch arm64 -std=gnu99
C99FLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CPICFLAGS = -fPIC
CPPFLAGS = -I/opt/R/arm64/include
C_VISIBILITY =

## C++ compilers
CXX = clang++ -arch arm64 -std=gnu++17
CXXFLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CXXPICFLAGS = -fPIC
CXX11 = clang++ -arch arm64
CXX11FLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CXX11PICFLAGS = -fPIC
CXX11STD = -std=gnu++11
CXX14 = clang++ -arch arm64
CXX14FLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CXX14PICFLAGS = -fPIC
CXX14STD = -std=gnu++14
CXX17 = clang++ -arch arm64
CXX17FLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CXX17PICFLAGS = -fPIC
CXX17STD = -std=gnu++17
CXX20 = clang++ -arch arm64
CXX20FLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CXX20PICFLAGS = -fPIC
CXX20STD = -std=gnu++20
CXX23 = clang++ -arch arm64
CXX23FLAGS = -falign-functions=64 -Wall -g -O2 $(LTO)
CXX23PICFLAGS = -fPIC
CXX23STD = -std=gnu++2b
CXX_VISIBILITY =

DYLIB_EXT = .dylib
DYLIB_LD = $(CC)
DYLIB_LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(CFLAGS) $(CPICFLAGS)
## should be used by packages for dylibs, but at the time of writing was not
DYLIB_LINK = $(DYLIB_LD) $(DYLIB_LDFLAGS) $(LDFLAGS)
ECHO = echo
ECHO_C = \c
ECHO_N =
ECHO_T =

## Fortran
FC = /opt/gfortran/bin/gfortran -arch arm64
FCFLAGS = -Wall -g -O2 $(LTO_FC)
## additional libs needed when linking with $(FC), e.g. on some Oracle compilers
FCLIBS_XTRA =
FFLAGS = -Wall -g -O2 $(LTO_FC)
FLIBS =  -L/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0 -L/opt/gfortran/lib -lgfortran -lemutls_w -lquadmath
FPICFLAGS = -fPIC
FPIEFLAGS = -fPIE
F_VISIBILITY =

FOUNDATION_CPPFLAGS =
FOUNDATION_LIBS = -framework Foundation
JAR = /usr/bin/jar
JAVA = /usr/bin/java
JAVAC = /usr/bin/javac
JAVAH = /usr/bin/javah
## JAVA_HOME might be used in the next three.
## They are for packages 'JavaGD' and 'rJava'
JAVA_HOME = /Library/Java/JavaVirtualMachines/jdk-11.0.18+10/Contents/Home
JAVA_CPPFLAGS = -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/darwin
JAVA_LIBS = -L$(JAVA_HOME)/lib/server -ljvm
JAVA_LD_LIBRARY_PATH = $(JAVA_HOME)/lib/server
LAPACK_LIBS = -L"$(R_HOME)/lib$(R_ARCH)" -lRlapack
LDFLAGS = -L/opt/R/arm64/lib
## we only need this is if it is external, as otherwise link to R
LIBINTL=   -Wl,-framework -Wl,CoreFoundation
LIBM =
LIBR0 = -L"$(R_HOME)/lib$(R_ARCH)"
LIBR1 = -lR
LIBR = -F/Library/Frameworks/R.framework/.. -framework R
LIBS = -L/opt/R/arm64/lib -lpcre2-8 -llzma -lbz2 -lz -licucore -ldl -lm -liconv
## needed by R CMD config
LIBnn = lib
LIBTOOL = $(SHELL) "$(R_HOME)/bin/libtool"
## set for --enable-lto, --enable-lto=check
LTO =
LTO_FC =
LTO_LD =
## set for --enable-lto=R, used in INSTALL
LTO_OPT =
LTO_FC_OPT =
## needed to build applications linking to static libR
MAIN_LD = $(CC)
MAIN_LDFLAGS =
MAIN_LINK = $(MAIN_LD) $(MAIN_LDFLAGS) $(LDFLAGS)
MKINSTALLDIRS = "$(R_HOME)/bin/mkinstalldirs"
NM = /usr/bin/nm -B
OBJC = clang -arch arm64
OBJCFLAGS = -falign-functions=64 -Wall -g -O2 -fobjc-exceptions $(LTO)
OBJC_LIBS = -lobjc
OBJCXX = clang++ -arch arm64
R_ARCH =
RANLIB = ranlib
SAFE_FFLAGS = -Wall -g -O2
SED = /usr/bin/sed
SHELL = /bin/sh
SHLIB_CFLAGS =
SHLIB_CXXFLAGS =
SHLIB_CXXLD = $(CXX)
SHLIB_CXXLDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(CXXFLAGS) $(CXXPICFLAGS) $(LTO_LD)
SHLIB_CXX11LD = $(CXX11) $(CXX11STD)
SHLIB_CXX11LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(CXX11FLAGS) $(CXX11PICFLAGS) $(LTO_LD)
SHLIB_CXX14LD = $(CXX14) $(CXX14STD)
SHLIB_CXX14LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(CXX14FLAGS) $(CXX14PICFLAGS) $(LTO_LD)
SHLIB_CXX17LD = $(CXX17) $(CXX17STD)
SHLIB_CXX17LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(CXX17FLAGS) $(CXX17PICFLAGS) $(LTO_LD)
SHLIB_CXX20LD = $(CXX20) $(CXX20STD)
SHLIB_CXX20LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(CXX20FLAGS) $(CXX20PICFLAGS) $(LTO_LD)
SHLIB_CXX23LD = $(CXX23) $(CXX23STD)
SHLIB_CXX23LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(CXX23FLAGS) $(CXX23PICFLAGS) $(LTO_LD)
SHLIB_EXT = .so
SHLIB_FCLD = $(FC)
SHLIB_FCLDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(FCFLAGS) $(FPICFLAGS)
SHLIB_FFLAGS =
SHLIB_LD = $(CC)
## used for add-on packages linked by C
SHLIB_LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(CFLAGS) $(CPICFLAGS) $(LTO_LD)
## used for standard packages
SHLIB_LDFLAGS_R = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup# $(CFLAGS) $(CPICFLAGS) $(LTO_LD)
SHLIB_LIBADD =
## We want to ensure libR is picked up from $(R_HOME)/lib
## before e.g. /usr/local/lib if a version is already installed.
SHLIB_LINK = $(SHLIB_LD) $(SHLIB_LDFLAGS) $(LIBR0) $(LDFLAGS)
SHLIB_OPENMP_CFLAGS =
SHLIB_OPENMP_CXXFLAGS =
SHLIB_OPENMP_FFLAGS =
STRIP_STATIC_LIB = strip -S
STRIP_SHARED_LIB = strip -x
TCLTK_CPPFLAGS = -I/opt/R/arm64/include -I/opt/R/arm64/include  -I/usr/X11R6/include
TCLTK_LIBS = -L/opt/R/arm64/lib -ltcl8.6 -L/opt/R/arm64/lib -ltk8.6 -L/usr/X11R6/lib -lX11 -Wl,-weak-lXss -lXext
YACC = bison -y

## for linking to libR.a
STATIC_LIBR = #  "$(R_HOME)/lib$(R_ARCH)/libR.a"  $(BLAS_LIBS) $(FLIBS)  $(LIBINTL) -lreadline -lncurses  $(LIBS)

## These are recorded as macros for legacy use in packages
## set on AIX, formerly for old glibc (-D__NO_MATH_INLINES)
R_XTRA_CFLAGS =
##  was formerly set on HP-UX
R_XTRA_CPPFLAGS =  -I"$(R_INCLUDE_DIR)" -DNDEBUG
## currently unset
R_XTRA_CXXFLAGS =
## used for gfortran >= 7 in R 3.6.1 - 4.2.x
R_XTRA_FFLAGS =

## SHLIB_CFLAGS SHLIB_CXXFLAGS SHLIB_FFLAGS are apparently currently unused
## SHLIB_CXXFLAGS is undocumented, there is no SHLIB_FCFLAGS
ALL_CFLAGS =  $(PKG_CFLAGS) $(CPICFLAGS) $(SHLIB_CFLAGS) $(CFLAGS)
## XDEFS is a temporary measure for use by R CMD INSTALL
ALL_CPPFLAGS =  -I"$(R_INCLUDE_DIR)" -DNDEBUG $(PKG_CPPFLAGS) $(CLINK_CPPFLAGS) $(CPPFLAGS)  $(XDEFS)
ALL_CXXFLAGS =  $(PKG_CXXFLAGS) $(CXXPICFLAGS) $(SHLIB_CXXFLAGS) $(CXXFLAGS)
ALL_OBJCFLAGS = $(PKG_OBJCFLAGS) $(CPICFLAGS) $(SHLIB_CFLAGS) $(OBJCFLAGS)
ALL_OBJCXXFLAGS = $(PKG_OBJCXXFLAGS) $(CXXPICFLAGS) $(SHLIB_CXXFLAGS) $(OBJCXXFLAGS)
ALL_FFLAGS =  $(PKG_FFLAGS) $(FPICFLAGS) $(SHLIB_FFLAGS) $(FFLAGS)
## can be overridden by R CMD SHLIB
P_FCFLAGS = $(PKG_FFLAGS)
ALL_FCFLAGS =  $(P_FCFLAGS) $(FPICFLAGS) $(SHLIB_FFLAGS) $(FCFLAGS)
## LIBR here as a couple of packages use this without SHLIB_LINK
ALL_LIBS = $(PKG_LIBS) $(SHLIB_LIBADD) $(LIBR) $(LIBINTL)

.SUFFIXES:
.SUFFIXES: .c .cc .cpp .d .f .f90 .f95 .m .mm .M .o

.c.o:
    $(CC) $(ALL_CPPFLAGS) $(ALL_CFLAGS) -c $< -o $@
.c.d:
    @echo "making $@ from $<"
    @$(CC) -MM $(ALL_CPPFLAGS) $< > $@
.cc.o:
    $(CXX) $(ALL_CPPFLAGS) $(ALL_CXXFLAGS) -c $< -o $@
.cpp.o:
    $(CXX) $(ALL_CPPFLAGS) $(ALL_CXXFLAGS) -c $< -o $@
.cc.d:
    @echo "making $@ from $<"
    @$(CXX) -M $(ALL_CPPFLAGS) $< > $@
.cpp.d:
    @echo "making $@ from $<"
    @$(CXX) -M $(ALL_CPPFLAGS) $< > $@
.m.o:
    $(OBJC) $(ALL_CPPFLAGS) $(ALL_OBJCFLAGS) -c $< -o $@
.m.d:
    @echo "making $@ from $<"
    @$(OBJC) -MM $(ALL_CPPFLAGS) $< > $@
.mm.o:
    $(OBJCXX) $(ALL_CPPFLAGS) $(ALL_OBJCXXFLAGS) -c $< -o $@
.M.o:
    $(OBJCXX) $(ALL_CPPFLAGS) $(ALL_OBJCXXFLAGS) -c $< -o $@
.f.o:
    $(FC) $(ALL_FFLAGS) -c $< -o $@
.f95.o:
    $(FC) $(ALL_FCFLAGS) -c  $< -o $@
.f90.o:
    $(FC) $(ALL_FCFLAGS) -c  $< -o $@

[bbolker]

Does this help at all https://github.com/RGLab/Rtsne.multicore/issues/2#issuecomment-300259543 ? (OpenMP enabled in compilation but maybe not found in load library path?)

[jaganmn]

@mikeroswell It seems like you obtained your R 4.3 binary from CRAN. In that case, you should not be using tools, libraries, or headers obtained from a Homebrew distribution. Doing so is bound to infect your .libPaths() with incompatible R package binaries.

My advice is to delete your existing Makevars, remove all packages installed from sources using that Makevars (or, to be safe, all packages in your user library), then start over following the instructions here which should be valid for R 4.3 even on macOS Sonoma. These recapitulate the instructions at https://mac.r-project.org/ which are maintained by the same person who maintains R for macOS. (The latter are the primary source, but include information about building R itself from sources that you don't really need.)

Sorry if the above seems extreme, but it is to me the best way to unmuddy the water here. I'm happy to join a call to walk you through the steps if needed.

[mikeroswell]

@jaganmn , I might take you up on your offer to walk through steps in real time soon, but first I'll ask for a bit more asynchronous troubleshooting.

I got partway:

> dyn.load("omp_test.so"); invisible(.Call("omp_test"))
OpenMP threads available: 10

But still got similar errors installing Rcpp, TMB from source.

Do I need to remove some Homebrew library installations from my $PATH? Where / how would I want to do that (if so)?

mikeroswell@Michaels-MBP ~ % echo $PATH
/opt/homebrew/opt/llvm/bin:/opt/homebrew/opt/llvm/bin:/opt/homebrew/opt/jpeg/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Library/TeX/texbin:/usr/local/git/bin

[mikeroswell]

I found ~/.zshrc and /etc/profile and deleted homebrew stuff from $PATH. I checked /etc/paths and don't see it there.

mikeroswell@Michaels-MBP egfR0 % echo $PATH
/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Library/TeX/texbin:/usr/local/git/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/mikeroswell/Applications/quarto/bin:/usr/texbin:/Applications/RStudio.app/Contents/Resources/app/quarto/bin:/Applications/RStudio.app/Contents/Resources/app/bin/postback

[jaganmn]

Yes, good catch. Homebrew is still masking the native C/C++ compilers on your PATH. R CMD CONFIG CC and R CMD CONFIG CXX should corroborate that.

I don't remember how Homebrew sets PATH. Have you also checked the files under /etc/paths.d?

[mikeroswell]

I just checked those, nothing looked like it was a brew installation. If you have suggestions on next steps or want to try to do this in real time, lmk! Thanks so so much, @jaganmn

[jaganmn]

I just tried installing Homebrew. It installed entirely under /opt/homebrew and asked me to run a script to append 2 lines to my ~/.bash_profile and set PATH in the current shell process. Just removing the 2 lines from ~/.bash_profile does not revert PATH in the current shell process because ~/.bash_profile is only run at startup. But if I exit the current shell and launch a new one, then in the new shell PATH is "normal" (without Homebrew).

So my expectation is that you should "only" need to modify your ~/.zprofile (if you use zsh) and restart Terminal.

If Homebrew remains on your PATH after all of that, or if your PATH is fixed but the compilation errors persist, then I'd want to see the (new) output of R CMD INSTALL.

[mikeroswell]

Thanks Mikael. PATH seems brew-free but still having TMB installation issue. latest_TMB_install.txt

[mikeroswell]

And latest_Rcpp_install.txt Rcpp**

[jaganmn]

Another guess, before you give up and ask R-SIG-Mac:

Uninstall your very new version of Xcode and then install Command Line Tools for Xcode 14.3.1 which is what I am using and closer to what CRAN used to build the R binary that you installed. The manual says:

Installing source packages which do not contain compiled code should work with no additional tools. For others you will need the 'Command Line Tools' for Xcode and compilers which match those used to build R ...

So we should comply.

[mikeroswell]

Well, I can't do that with this version of macOS; I'll post to the mailing list.

[bbolker]

A 14000+-line error log is impressive ...

[jaganmn]

Oops, mental lapse ... The compatibility is in the other direction: I can install Xcode 15 on macOS Ventura. Perhaps if I do that then I will be able to reproduce the errors. I will experiment (later).

[jaganmn]

So we are seeing preprocessor errors like

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/cstddef:46:5: error:
      <cstddef> tried including <stddef.h> but didn't find libc++'s <stddef.h>
      header. This usually means that your header search paths are not
      configured properly. The header search paths should contain the C++
      Standard Library headers before any C Standard Library, and you are
      probably using compiler flags that make that not be the case.

emitted from headers like cstddef in the macOS 14.2 SDK bundled with Xcode 15.1.

cstddef includes stddef.h and expects the preprocessor to find the version of stddef.h in the C++ Standard Library. But the preprocessor is finding the version of stddef.h in the C Standard Library. That suggests that we are prepending to the header search path a directory that contains (but really shouldn't contain) stddef.h.

The culprit, judging from the command line in the Rcpp installation output, would be one of these directories:

/Library/Frameworks/R.framework/Resources/include
/opt/R/arm64/include
/usr/local/include

The most likely candidate in my mind is /usr/local/include, because /usr/local is a standard installation prefix and would have been used by Homebrew on Apple silicon before it introduced support for native installations under /opt/homebrew.

Hence @mikeroswell may need to sudo rm -rf /usr/local/* if he sees Homebrew remnants there and especially if he sees that /usr/local/include contains stddef.h and other headers from the C Standard Library ...

[mikeroswell]

Working now!

I removed lots of stuff:

sudo rm -rf /usr/local/*
 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/uninstall.sh)"
sudo rm -rf /opt/homebrew
sudo rm -rf /usr/local/Homebrew

Then I reinstalled the GNU FORTRAN comnpiler, openMP, and R.

Then, I installed the R packages Rcpp, Matrix, and TMB from source.

I was then able to install epigrowthfit.

Thanks for all the help, @jaganmn !

[mikeroswell]

@bbolker , Should I in some way close this issue or do other housekeeping? It seems like the upshot was: one way around my issue was to remove all the homebrew installations and try again; I feel like someone encountering epigrowthfit in the wild cd take a long time to discern that from this thread...

[jaganmn]

I'm in the process of updating the README. I'll add a note mentioning perils with Homebrew. I think that we can leave this issue open for future installation problems.

I'll just add that you can use Homebrew to manage your software. You just need to make sure that your Homebrew stuff doesn't infect your R stuff if you've obtained R from CRAN. Don't install under /usr/local with Homebrew (non-issue now that Homebrew installs under /opt/homebrew on Apple silicon) and don't use Makevars to involve Homebrew in R package installation.

[davidearn]

After the README is done, I might upgrade my OS to Sonoma and check the instructions are adequate. It is good that it was possible to get everything to work, but having to obliterate /usr/local/* is a huge pain. We don't want users to have to do that in order to install epigrowthfit.

[jaganmn]

It is an issue primarily for people who have decided to install LLVM and mask Apple's Command Line Tools, i.e., not really unique to Sonoma. The README will mention that Homebrew users can start with brew uninstall llvm, but that won't catch all of the ways that the user could have contaminated their search paths. It doesn't need to be too comprehensive, though, because ultimately CRAN will provide binaries for Windows and macOS.

[bbolker]

Another possible solution/assistance for installers will be to use r-universe to provide binaries ...