search

stan-dev / rstan

RStan, the R interface to Stan
https://mc-stan.org
1.04k stars 268 forks source link

Stan isn't working any more in R on Mac (High Sierra) #799

Open kgoldfeld opened 4 years ago

kgoldfeld commented 4 years ago

LIke many other folks, my stan environment in R is broken. I am on a Mac running High Sierra (Version 10.13.6), R version 4.0.2, latest version of Rstudio, and I rstan version 2.21.1. I have removed and reinstalled rstan several times, and also udpated stanHeaders.

The problem start when I try translate my model spec to C++ using rstan::stanc:

rt <- stanc("Programs/lm.stan")

This is the message I get:

clang-10: error: no such file or directory: 'NYU'
clang-10: error: no such file or directory: 'Langone'
clang-10: error: no such file or directory: 'Health/R'
clang-10: error: no such file or directory: 'Projects/Covid-19'
clang-10: error: no such file or directory: 'plasma'
clang-10: error: cannot specify -o when generating multiple output files
The NEXT version of Stan will not be able to pre-process your Stan program.
Please open an issue at
 https://github.com/stan-dev/stanc3/issues 
if you can share or at least describe your Stan program. This will help ensure that Stan
continues to work on your Stan programs in the future. Thank you!
This message can be avoided by wrapping your function call inside suppressMessages().
Warning message:
In file.remove(c(unprocessed, processed)) :
  cannot remove file '/var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T//Rtmp1Q9MuJ/filed22d10d067dc.stan', reason 'No such file or directory'

Any suggestions?

I don't think it has anything to do with my specific stan code, since I get this with every stan file I attempt to translate. In this case, it is a super simple regression model:

data {
  int<lower=0> N;
  vector[N] x;
  vector[N] y;
}

parameters {
  real b0;
  real b1;
  real<lower=0> sigma;
}

model {
  y ~ normal(b0 + b1*x, sigma);
}
SteveBronder commented 4 years ago

Thanks for submitting an issue! Running this on the latest cmdstan with the new compiler the model does compile so I think this is an rstan issue (in particular it looks like it might be something with how CXX is set to clang-10). Hope you don't mind I'm going to move it over to there

SteveBronder commented 4 years ago

Can you post your Makevars?

kgoldfeld commented 4 years ago

I'd be happy to - but can you tell me how?

kgoldfeld commented 4 years ago

I found it, and I recall I had to edit it recently:

# Settings from /etc/R/Makeconf with "non-portable flag(s):"
# ‘-Wdate-time’ ‘-Werror=format-security’ ‘-Wformat’ replaced by -Wall -pedantic
# and without -fdebug-prefix-map=... 

# Original file

# CFLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2 $(LTO)
# CXXFLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2 $(LTO)
# CXX98FLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2
# CXX11FLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2
# CXX14FLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2

# Added 2020-06-09 
# https://ryanhomer.github.io/posts/build-openmp-macos-catalina-complete

XCBASE:=$(shell xcrun --show-sdk-path)
LLVMBASE:=$(shell brew --prefix llvm)
GCCBASE:=$(shell brew --prefix gcc)
GETTEXT:=$(shell brew --prefix gettext)

# All seem to work

CXX11=$(LLVMBASE)/bin/clang++
CXX14=$(LLVMBASE)/bin/clang++
CXX17=$(LLVMBASE)/bin/clang++
CXX1X=$(LLVMBASE)/bin/clang++

FC=$(GCCBASE)/bin/gfortran
F77=$(GCCBASE)/bin/gfortran
FLIBS=-L$(GCCBASE)/lib/gcc/9/ -lm

# Added 2020-06-18 for data.table issues
# https://github.com/Rdatatable/data.table/wiki/Installation#openmp-enabled-compiler-for-mac

LLVM_LOC = /usr/local/opt/llvm

CC=$(LLVM_LOC)/bin/clang -fopenmp
# CC=$(LLVMBASE)/bin/clang  # does not work
CXX=$(LLVM_LOC)/bin/clang++ -fopenmp
## CXX=$(LLVMBASE)/bin/clang++ # works also

# -O3 should be faster than -O2 (default) level optimisation ..

CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe

LDFLAGS=-L"$(LLVMBASE)/lib" -L"$(GETTEXT)/lib" --sysroot="$(XCBASE)"
# Next generates warning, but seems to work
## LDFLAGS=-L/usr/local/opt/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib

CPPFLAGS=-isystem "$(LLVMBASE)/include" -isysroot "$(XCBASE)"
# Next seems fine as well
## CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include
SteveBronder commented 4 years ago

Oh sorry I assumed by seeing clang-10 you had modified your Makevars file. I thought clang-7 was the default clang for mac

It's most likely in ~/.R/Makevars and you can print it from the mac terminal with cat ~/.R/Makevars. but if you haven't touched it then that may not be the issue.

kgoldfeld commented 4 years ago

I just sent it - and I did edit it at some point ...

# Settings from /etc/R/Makeconf with "non-portable flag(s):"
# ‘-Wdate-time’ ‘-Werror=format-security’ ‘-Wformat’ replaced by -Wall -pedantic
# and without -fdebug-prefix-map=... 

# Original file

# CFLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2 $(LTO)
# CXXFLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2 $(LTO)
# CXX98FLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2
# CXX11FLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2
# CXX14FLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2

# Added 2020-06-09 
# https://ryanhomer.github.io/posts/build-openmp-macos-catalina-complete

XCBASE:=$(shell xcrun --show-sdk-path)
LLVMBASE:=$(shell brew --prefix llvm)
GCCBASE:=$(shell brew --prefix gcc)
GETTEXT:=$(shell brew --prefix gettext)

# All seem to work

CXX11=$(LLVMBASE)/bin/clang++
CXX14=$(LLVMBASE)/bin/clang++
CXX17=$(LLVMBASE)/bin/clang++
CXX1X=$(LLVMBASE)/bin/clang++

FC=$(GCCBASE)/bin/gfortran
F77=$(GCCBASE)/bin/gfortran
FLIBS=-L$(GCCBASE)/lib/gcc/9/ -lm

# Added 2020-06-18 for data.table issues
# https://github.com/Rdatatable/data.table/wiki/Installation#openmp-enabled-compiler-for-mac

LLVM_LOC = /usr/local/opt/llvm

CC=$(LLVM_LOC)/bin/clang -fopenmp
# CC=$(LLVMBASE)/bin/clang  # does not work
CXX=$(LLVM_LOC)/bin/clang++ -fopenmp
## CXX=$(LLVMBASE)/bin/clang++ # works also

# -O3 should be faster than -O2 (default) level optimisation ..

CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe

LDFLAGS=-L"$(LLVMBASE)/lib" -L"$(GETTEXT)/lib" --sysroot="$(XCBASE)"
# Next generates warning, but seems to work
## LDFLAGS=-L/usr/local/opt/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib

CPPFLAGS=-isystem "$(LLVMBASE)/include" -isysroot "$(XCBASE)"
# Next seems fine as well
## CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include
SteveBronder commented 4 years ago

Ah ic so it looks like you have a lot of stuff here setup for data.table. Can you try commenting out all that stuff and trying again? You may need to do a fresh install of rstan. Your error clang-10: error: no such file or directory: 'NYU' seems to be coming directly from clang so my guess is that this could be something like missing quotes from somewhere in your makevars. Also we rely on C++14 so I'm surprised that CXXFLAGS works specifying std=c++11

kgoldfeld commented 4 years ago

I will do that, but what will happen to the setup I did for data.table?

kgoldfeld commented 4 years ago

OK - I reverted back to the old version of Makevars:

# Original file

CFLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2 $(LTO)
CXXFLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2 $(LTO)
CXX98FLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2
CXX11FLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2
CXX14FLAGS = -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2

And I removed and reinstalled rstan. And I ran stanc again. I am no longer getting those clang-10 error messages, so that is good. But, I am getting this:

sh: clang++ -mmacosx-version-min=10.13: command not found
The NEXT version of Stan will not be able to pre-process your Stan program.
Please open an issue at
 https://github.com/stan-dev/stanc3/issues 
if you can share or at least describe your Stan program. This will help ensure that Stan
continues to work on your Stan programs in the future. Thank you!
This message can be avoided by wrapping your function call inside suppressMessages().
Warning messages:
1: In system2(CXX, args = ARGS) : error in running command
2: In file.remove(c(unprocessed, processed)) :
  cannot remove file '/var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T//Rtmp7qZ6QY/filee74376eadec4.stan', reason 'No such file or directory'
SteveBronder commented 4 years ago

Progress! So for that check out #774 it looks like you can fix that by installing the latest version on github (it has to do with how get_CXX() was returning back clang++ -mmacosx-version-min=10.13 instead of just clang++)

install.packages("remotes")
remotes::install_github("stan-dev/rstan", ref = "develop", subdir = "rstan/rstan")
kgoldfeld commented 4 years ago

I got a clang error message on the install:

> remotes::install_github("stan-dev/rstan", ref = "develop", subdir = "rstan/rstan")
Downloading GitHub repo stan-dev/rstan@develop
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules --branch develop https://github.com/stan-dev/stan.git /var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T//RtmpWKfPuS/remotesfa4a19712297/stan-dev-rstan-74f847c/rstan/rstan/../../StanHeaders/inst/include/upstream
Cloning into '/var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T//RtmpWKfPuS/remotesfa4a19712297/stan-dev-rstan-74f847c/rstan/rstan/../../StanHeaders/inst/include/upstream'...
Checking out files: 100% (2240/2240), done.
Submodule 'lib/stan_math' (https://github.com/stan-dev/math.git) registered for path 'lib/stan_math'
Cloning into '/private/var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T/RtmpWKfPuS/remotesfa4a19712297/stan-dev-rstan-74f847c/StanHeaders/inst/include/upstream/lib/stan_math'...
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules --branch develop https://github.com/stan-dev/math.git /var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T//RtmpWKfPuS/remotesfa4a19712297/stan-dev-rstan-74f847c/rstan/rstan/../../StanHeaders/inst/include/mathlib
Cloning into '/var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T//RtmpWKfPuS/remotesfa4a19712297/stan-dev-rstan-74f847c/rstan/rstan/../../StanHeaders/inst/include/mathlib'...
Checking out files: 100% (25928/25928), done.
These packages have more recent versions available.
It is recommended to update all of them.
Which would you like to update?

1: All                          
2: CRAN packages only           
3: None                         
4: vctrs (0.3.1 -> 0.3.2) [CRAN]

Enter one or more numbers, or an empty line to skip updates:
3
✓  checking for file ‘/private/var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T/RtmpWKfPuS/remotesfa4a19712297/stan-dev-rstan-74f847c/rstan/rstan/DESCRIPTION’ (738ms)
─  preparing ‘rstan’: (5.1s)
✓  checking DESCRIPTION meta-information ...
─  cleaning src
─  checking for LF line-endings in source and make files and shell scripts (958ms)
─  checking for empty or unneeded directories (3.1s)
─  looking to see if a ‘data/datalist’ file should be added
─  building ‘rstan_2.21.2.tar.gz’

* installing *source* package ‘rstan’ ...
** using staged installation
** libs
clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"../inst/include" -I"../inst/include/boost_not_in_BH" -I"." -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION -DBOOST_NO_AUTO_PTR -D_REENTRANT -DSTAN_THREADS   See you next time ... Thu Jul 16 17:13:07 2020  -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/RcppEigen/include' -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/BH/include' -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/StanHeaders/include' -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/RcppParallel/include' -I/usr/local/include   -fPIC  -g -O2 -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2 -c Module.cpp -o Module.o
clang: error: no such file or directory: 'See'
clang: error: no such file or directory: 'you'
clang: error: no such file or directory: 'next'
clang: error: no such file or directory: 'time'
clang: error: no such file or directory: '...'
clang: error: no such file or directory: 'Thu'
clang: error: no such file or directory: 'Jul'
clang: error: no such file or directory: '16'
clang: error: no such file or directory: '17:13:07'
clang: error: no such file or directory: '2020'
make: *** [Module.o] Error 1
ERROR: compilation failed for package ‘rstan’
* removing ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library/rstan’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library/rstan’
Error: Failed to install 'rstan' from GitHub:
  (converted from warning) installation of package ‘/var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T//RtmpWKfPuS/filefa4a14e220b/rstan_2.21.2.tar.gz’ had non-zero exit status
SteveBronder commented 4 years ago

Ah,

clang++ (...stuff...) -DSTAN_THREADS   See you next time ... Thu Jul 16 17:13:07 2020  -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include'

Do you have spaces in your folder path? I'd remove those spaces or look at #776 which has a similar problem with using spaces in renv.

kgoldfeld commented 4 years ago

I haven't had much luck - not sure exactly what folder path is causing the problem. I did set the cache path using

Sys.setenv(RENV_PATHS_CACHE ="~/Library/Frameworks/renv/cache")

and then I installed StanHeaders (after removing it), which worked fine:

renv::install("StanHeaders")

And then I tried to install the development version of rstan as before. But I am still getting the same error message:

clang: error: no such file or directory: 'See'
clang: error: no such file or directory: 'you'
clang: error: no such file or directory: 'next'
clang: error: no such file or directory: 'time'
clang: error: no such file or directory: '...'
clang: error: no such file or directory: 'Thu'
clang: error: no such file or directory: 'Jul'
clang: error: no such file or directory: '16'
clang: error: no such file or directory: '18:36:56'
clang: error: no such file or directory: '2020'
make: *** [Module.o] Error 1
ERROR: compilation failed for package ‘rstan’
* removing ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library/rstan’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library/rstan’
Error: Failed to install 'rstan' from GitHub:
  (converted from warning) installation of package ‘/var/folders/wt/rrrkt68n08b0jrstl_87kpkc0000gn/T//Rtmp5b7NKT/file104252a35c17e/rstan_2.21.2.tar.gz’ had non-zero exit status
kgoldfeld commented 4 years ago

Well - I've admitted defeat, and was able to reinstall an earlier version of rstan. That is working fine, so I guess I'll stick with that. Thanks for trying.