Open rok-cesnovar opened 3 years ago
so, the stanc() function seems? broken, in that it requires a file argument when, if you use the model_code argument, file should not be needed.
so, the stanc() function seems? broken, in that it requires a file argument when, if you use the model_code argument, file should not be needed.
stanc()
interface and arguments didn't change. Can you provide a reproducible example?
library(rstan)
st <- '
functions{
real plus(real a){
return a + 1;
}
}
'
testcpp <- stanc(model_code = st)
Updated:
remove.packages(c("StanHeaders", "rstan")) install.packages("StanHeaders", repos = c("https://mc-stan.org/r-packages/", getOption("repos"))) install.packages("rstan", repos = c("https://mc-stan.org/r-packages/", getOption("repos")))
I installed using the above and then tried the following on my system (session info included below). This now seems to work :+1: Thank you @hsbadr ! Note that this is with the "broken" CRAN version of RcppParallel installed (see session info).
library(rstan)
#> Warning: package 'rstan' was built under R version 4.0.3
#> Loading required package: StanHeaders
#> Warning: package 'StanHeaders' was built under R version 4.0.3
#> rstan (Version 2.26.0.9000, GitRev: 2e1f913d3ca3)
#> For execution on a local, multicore CPU with excess RAM we recommend calling
#> options(mc.cores = parallel::detectCores()).
#> To avoid recompilation of unchanged Stan programs, we recommend calling
#> rstan_options(auto_write = TRUE)
#> Do not specify '-march=native' in 'LOCAL_CPPFLAGS' or a Makevars file
m <- stan_model(model_code = 'parameters {real y;} model {y ~ normal(0,1);}')
f <- sampling(m, iter = 100)
#>
#> SAMPLING FOR MODEL 'anon_model' NOW (CHAIN 1).
#> Chain 1:
#> Chain 1: Gradient evaluation took 1.1e-05 seconds
#> Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0.11 seconds.
#> Chain 1: Adjust your expectations accordingly!
#> Chain 1:
#> Chain 1:
#> Chain 1: WARNING: There aren't enough warmup iterations to fit the
#> Chain 1: three stages of adaptation as currently configured.
#> Chain 1: Reducing each adaptation stage to 15%/75%/10% of
#> Chain 1: the given number of warmup iterations:
#> Chain 1: init_buffer = 7
#> Chain 1: adapt_window = 38
#> Chain 1: term_buffer = 5
#> Chain 1:
#> Chain 1: Iteration: 1 / 100 [ 1%] (Warmup)
#> Chain 1: Iteration: 10 / 100 [ 10%] (Warmup)
#> Chain 1: Iteration: 20 / 100 [ 20%] (Warmup)
#> Chain 1: Iteration: 30 / 100 [ 30%] (Warmup)
#> Chain 1: Iteration: 40 / 100 [ 40%] (Warmup)
#> Chain 1: Iteration: 50 / 100 [ 50%] (Warmup)
#> Chain 1: Iteration: 51 / 100 [ 51%] (Sampling)
#> Chain 1: Iteration: 60 / 100 [ 60%] (Sampling)
#> Chain 1: Iteration: 70 / 100 [ 70%] (Sampling)
#> Chain 1: Iteration: 80 / 100 [ 80%] (Sampling)
#> Chain 1: Iteration: 90 / 100 [ 90%] (Sampling)
#> Chain 1: Iteration: 100 / 100 [100%] (Sampling)
#> Chain 1:
#> Chain 1: Elapsed Time: 0 seconds (Warm-up)
#> Chain 1: 0 seconds (Sampling)
#> Chain 1: 0 seconds (Total)
#> Chain 1:
#>
#> SAMPLING FOR MODEL 'anon_model' NOW (CHAIN 2).
#> Chain 2:
#> Chain 2: Gradient evaluation took 3e-06 seconds
#> Chain 2: 1000 transitions using 10 leapfrog steps per transition would take 0.03 seconds.
#> Chain 2: Adjust your expectations accordingly!
#> Chain 2:
#> Chain 2:
#> Chain 2: WARNING: There aren't enough warmup iterations to fit the
#> Chain 2: three stages of adaptation as currently configured.
#> Chain 2: Reducing each adaptation stage to 15%/75%/10% of
#> Chain 2: the given number of warmup iterations:
#> Chain 2: init_buffer = 7
#> Chain 2: adapt_window = 38
#> Chain 2: term_buffer = 5
#> Chain 2:
#> Chain 2: Iteration: 1 / 100 [ 1%] (Warmup)
#> Chain 2: Iteration: 10 / 100 [ 10%] (Warmup)
#> Chain 2: Iteration: 20 / 100 [ 20%] (Warmup)
#> Chain 2: Iteration: 30 / 100 [ 30%] (Warmup)
#> Chain 2: Iteration: 40 / 100 [ 40%] (Warmup)
#> Chain 2: Iteration: 50 / 100 [ 50%] (Warmup)
#> Chain 2: Iteration: 51 / 100 [ 51%] (Sampling)
#> Chain 2: Iteration: 60 / 100 [ 60%] (Sampling)
#> Chain 2: Iteration: 70 / 100 [ 70%] (Sampling)
#> Chain 2: Iteration: 80 / 100 [ 80%] (Sampling)
#> Chain 2: Iteration: 90 / 100 [ 90%] (Sampling)
#> Chain 2: Iteration: 100 / 100 [100%] (Sampling)
#> Chain 2:
#> Chain 2: Elapsed Time: 0 seconds (Warm-up)
#> Chain 2: 0 seconds (Sampling)
#> Chain 2: 0 seconds (Total)
#> Chain 2:
#>
#> SAMPLING FOR MODEL 'anon_model' NOW (CHAIN 3).
#> Chain 3:
#> Chain 3: Gradient evaluation took 2e-06 seconds
#> Chain 3: 1000 transitions using 10 leapfrog steps per transition would take 0.02 seconds.
#> Chain 3: Adjust your expectations accordingly!
#> Chain 3:
#> Chain 3:
#> Chain 3: WARNING: There aren't enough warmup iterations to fit the
#> Chain 3: three stages of adaptation as currently configured.
#> Chain 3: Reducing each adaptation stage to 15%/75%/10% of
#> Chain 3: the given number of warmup iterations:
#> Chain 3: init_buffer = 7
#> Chain 3: adapt_window = 38
#> Chain 3: term_buffer = 5
#> Chain 3:
#> Chain 3: Iteration: 1 / 100 [ 1%] (Warmup)
#> Chain 3: Iteration: 10 / 100 [ 10%] (Warmup)
#> Chain 3: Iteration: 20 / 100 [ 20%] (Warmup)
#> Chain 3: Iteration: 30 / 100 [ 30%] (Warmup)
#> Chain 3: Iteration: 40 / 100 [ 40%] (Warmup)
#> Chain 3: Iteration: 50 / 100 [ 50%] (Warmup)
#> Chain 3: Iteration: 51 / 100 [ 51%] (Sampling)
#> Chain 3: Iteration: 60 / 100 [ 60%] (Sampling)
#> Chain 3: Iteration: 70 / 100 [ 70%] (Sampling)
#> Chain 3: Iteration: 80 / 100 [ 80%] (Sampling)
#> Chain 3: Iteration: 90 / 100 [ 90%] (Sampling)
#> Chain 3: Iteration: 100 / 100 [100%] (Sampling)
#> Chain 3:
#> Chain 3: Elapsed Time: 0 seconds (Warm-up)
#> Chain 3: 0 seconds (Sampling)
#> Chain 3: 0 seconds (Total)
#> Chain 3:
#>
#> SAMPLING FOR MODEL 'anon_model' NOW (CHAIN 4).
#> Chain 4:
#> Chain 4: Gradient evaluation took 3e-06 seconds
#> Chain 4: 1000 transitions using 10 leapfrog steps per transition would take 0.03 seconds.
#> Chain 4: Adjust your expectations accordingly!
#> Chain 4:
#> Chain 4:
#> Chain 4: WARNING: There aren't enough warmup iterations to fit the
#> Chain 4: three stages of adaptation as currently configured.
#> Chain 4: Reducing each adaptation stage to 15%/75%/10% of
#> Chain 4: the given number of warmup iterations:
#> Chain 4: init_buffer = 7
#> Chain 4: adapt_window = 38
#> Chain 4: term_buffer = 5
#> Chain 4:
#> Chain 4: Iteration: 1 / 100 [ 1%] (Warmup)
#> Chain 4: Iteration: 10 / 100 [ 10%] (Warmup)
#> Chain 4: Iteration: 20 / 100 [ 20%] (Warmup)
#> Chain 4: Iteration: 30 / 100 [ 30%] (Warmup)
#> Chain 4: Iteration: 40 / 100 [ 40%] (Warmup)
#> Chain 4: Iteration: 50 / 100 [ 50%] (Warmup)
#> Chain 4: Iteration: 51 / 100 [ 51%] (Sampling)
#> Chain 4: Iteration: 60 / 100 [ 60%] (Sampling)
#> Chain 4: Iteration: 70 / 100 [ 70%] (Sampling)
#> Chain 4: Iteration: 80 / 100 [ 80%] (Sampling)
#> Chain 4: Iteration: 90 / 100 [ 90%] (Sampling)
#> Chain 4: Iteration: 100 / 100 [100%] (Sampling)
#> Chain 4:
#> Chain 4: Elapsed Time: 0 seconds (Warm-up)
#> Chain 4: 0 seconds (Sampling)
#> Chain 4: 0 seconds (Total)
#> Chain 4:
#> Warning: Bulk Effective Samples Size (ESS) is too low, indicating posterior means and medians may be unreliable.
#> Running the chains for more iterations may help. See
#> https://mc-stan.org/misc/warnings.html#bulk-ess
#> Warning: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable.
#> Running the chains for more iterations may help. See
#> https://mc-stan.org/misc/warnings.html#tail-ess
Created on 2021-01-20 by the reprex package (v0.3.0)
library(rstan) st <- ' functions{ real plus(real a){ return a + 1; } } ' testcpp <- stanc(model_code = st)
Ok. I'll look into it. It's likely b/c I added a preprocessor. Can you try: testcpp <- stanc(file = "", model_code = st)
for now?
That works and everything compiles and runs, and looks like normal, but the log probability / parameter values are all wrong. Will take me some time to see why.
That works and everything compiles and runs, and looks like normal, but the log probability / parameter values are all wrong. Will take me some time to see why.
In the current version I'm testing, it complains about plus
in your code. Need to rename it.
Yes, plus is a Stan function as well and user defined functions can not shadow Stan function names.
That was just a cheap mwe, not my actual test problem ;) but sure. I've narrowed down the problem to the gradient calculation of some limited set of functions, but the functions on their own seem ok... will try some more.
Additionally, I posted a (edit) very silly issue and closed here: https://github.com/stan-dev/stanc3/issues/801 but maybe a more informative error could be considered sometime :)
One further thing that could be done for testing is to use the revdepcheck package from RStudio et. al. to check whether this new version breaks any of the existing CRAN packages with dependencies on {rstan}. Unfortunately I don't think I have the time nor computing power to do it myself at the moment though.
@jeffreypullin I did that (in my environment: Linux + Intel compilers + R-devel) and I'm using this version in production with almost all of its dependencies. I think we've two tasks:
stanc
file
issue that I've fixed).The good thing is that this is now following Stan development and the package deployment has been automated. This can ensure that rstan
development is coupled with Stan.
Oh wow, sorry @hsbadr I didn't realize you'd already done that! I completely agree with 1 and 2.
The problem is the rstan tests arent really that rich. For example this case with the model code should have definitely been caught by at least one unit test. Expanding unit tests is definitely something we need to do in order to simplify the release cycle.
Ok. I spent the morning diagnosing the 'incorrect gradient under certain conditions' thing, and it magically fixed itself :/ No idea how / why, maybe some remnant left in memory? Anyway, tests are passing, seems fine, speed seems nicely bumped up!
One thing: in rstan 2.21, I got this:
Variable "JAX" does not exist.
error in 'model_ctsmgen' at line 1055, column 46
-------------------------------------------------
1053:
1054: asymDIFFUSION[ derrind, derrind ] =
1055: to_matrix( (add_diag( -sqkron_prod(JAX[ derrind, derrind ], JAX[ derrind, derrind ]),
but this parses in rstan 2.26 and then gives a compile error.
Can you paste the entire model or link it. Thanks
@cdriveraus @rok-cesnovar I think it's more of stanc
--allow-undefined
option. The variable JAX
seems to be undefined and in one version it's allowed in stanc
and failed in compilation and in the other one stanc
rejected` it. You may need to provide the headers that define it.
Not sure I follow. It was just a miscapitalisation (should have been JAx) .
Hm, I get
Semantic error in 'string', line 917, column 43 to column 46: Identifier 'JAX' not in scope.
Hm, I get
Semantic error in 'string', line 917, column 43 to column 46: Identifier 'JAX' not in scope.
I think this is likely a valid error and may depend on the compiler version and flags. Does it run with the old version of rsran
? If not, then, it should be in a separate issue, not related to this one.
Yes, this error I get is correct and valid.
yes that is a valid error. However, when I use stanc("", model_code=x) , it doesn't give the error, which is the problem.
yes that is a valid error. However, when I use stanc("", model_code=x) , it doesn't give the error, which is the problem.
It just allows undefined variables, not really an issue with rstan
as it's a valid error and stanc3
didn't complain. So, @rok-cesnovar, is there an option in stanc3
to check undefined variables and how to pass it to the JS?
Hm, I am not sure why would it allow undefined variables? If I place that model here: https://rok-cesnovar.github.io/stanc3js-demo/index.html I get the error. And this link uses the exact same javascript file. So the issue might be rstan does not catch the error? Will check.
if I use stanc(file=x) instead of the model_code approach, it gives the error, so I don't think this is intended (or at least, consistent!) behaviour.
What's the status (...$status
) of the generated code?
@cdriveraus @rok-cesnovar It does work as expected. Have you installed the latest rstan 2.26
binaries?
> library(rstan)
Loading required package: StanHeaders
rstan (Version 2.26.0.9000, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
options(mc.cores = parallel::detectCores()).
To avoid recompilation of unchanged Stan programs, we recommend calling
rstan_options(auto_write = TRUE)
> cppcode <- stanc(file = "ctsm.stan")
Info: Found int division at 'string', line 57, column 10 to column 17:
(d * d - d) / 2
Values will be rounded towards zero. If rounding is not desired you can write
the division as
(d * d - d) / 2.0
0
Semantic error in 'string', line 867, column 43 to column 46:
Identifier 'JAX' not in scope.
> cppcode$status
[1] FALSE
Maybe it's confusing because 'giving an error' is actually the desired behaviour. yes, stanc(file=x) works correctly. stanc(file="",model_code=x) does not return an error, which seems incorrect.
Maybe it's confusing because 'giving an error' is actually the desired behaviour. yes, stanc(file=x) works correctly. stanc(file="",model_code=x) does not return an error, which seems incorrect.
That's a hack to test the older version; don't use it 😄
And, yes, I can convert this to Error
and catch the hack too.
@cdriveraus @rok-cesnovar I've pushed a few commits to catch those cases and stop on stanc3
errors. Give it a try after it builds the binaries and let me know if it addresses the problem.
@cdriveraus @rok-cesnovar Here's the new behavior:
> cppcode <- stanc(file = "ctsm.stan")
Info: Found int division at 'string', line 57, column 10 to column 17:
(d * d - d) / 2
Values will be rounded towards zero. If rounding is not desired you can write
the division as
(d * d - d) / 2.0
Error in stanc(file = "ctsm.stan") : 0
Semantic error in 'string', line 867, column 43 to column 46:
Identifier 'JAX' not in scope.
and for your other example:
> st <- '
+ functions{
+ real plus(real a){
+ return a + 1;
+ }
+ }
+ '
>
> testcpp <- stanc(model_code = st)
Error in stanc(model_code = st) : 0
Semantic error in 'string', line 2, column 6 to column 10:
Identifier 'plus' clashes with Stan Math library function.
> testcpp <- stanc(file = "", model_code = st)
Error in stanc(file = "", model_code = st) : Empty or invalid filename!
I've been testing this on my machine and some of my scripts and I've run in to some interesting errors but I can't figure out if it's just my environment. It only occurs when I modify threads_per_chain to be >1, and won't compile any stan scripts if this is set. I noticed that it is adding some includes for parallel processing, but can't figure out what's not linking. Here's the error I'm getting when trying to run the reduce_sum example script.
C:/Users/Reece/Documents/R/win-library/4.0/StanHeaders/include/stan/math/prim/functor/reduce_sum.hpp:159:41: error: no matching function for call to 'parallel_deterministic_reduce(tbb::blocked_range<long long unsigned int>, stan::math::internal::reduce_sum_impl<modelf6c5db169a0__namespace::partial_sum_rsfunctor__, void, double, const std::vector<int>&, const Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&>::recursive_reducer&, tbb::simple_partitioner&)'
I'm also getting these errors,
"C:/Users/Reece/Documents/R/win-library/4.0/RcppParallel/include/tbb/task_scheduler_init.h:56:7: error: 'tbb::internal::no_copy::no_copy(const tbb::internal::no_copy&)' is private within this context"
"filef6c35717db5.cpp:17:49: error: use of deleted function 'tbb::task_scheduler_init::task_scheduler_init(const tbb::task_scheduler_init&)'"
`- Session info --------------------------------------------------------------------------------------------------------------------------
setting value
version R version 4.0.3 (2020-10-10)
os Windows 10 x64
system x86_64, mingw32
ui RStudio
language (EN)
collate English_United States.1252
ctype English_United States.1252
tz America/Chicago
date 2021-01-27
@Reece844 This looks like a TBB issue/conflict. I'm using an external oneTBB library for both RcppParallel
and math
, with the help of https://github.com/stan-dev/math/pull/2261 (task_scheduler_init.h
has been removed from the new TBB interface).
Can you share a reprex
with your model code? and maybe, test it on another machine, preferably Linux (just for env testing).
Also, does your model work with cmdstanr
? If you run into similar issues, then it's most likely due to your model or env.
This is just the basic reduce_sum example from the Stan reference page, but it gives the same error as my own model. I can look into that tomorrow.
model_c <-"functions {
real partial_sum(int[] y_slice,
int start, int end,
vector x,
vector beta) {
return bernoulli_logit_lpmf(y_slice | beta[1] + beta[2] * x[start:end]);
}
}
data {
int N;
int y[N];
vector[N] x;
}
parameters {
vector[2] beta;
}
model {
int grainsize = 1;
beta ~ std_normal();
target += reduce_sum(partial_sum, y,
grainsize,
x, beta);
}"
library(rstan)
#> Loading required package: StanHeaders
#> rstan (Version 2.26.0.9000, GitRev: 2e1f913d3ca3)
#> For execution on a local, multicore CPU with excess RAM we recommend calling
#> options(mc.cores = parallel::detectCores()).
#> To avoid recompilation of unchanged Stan programs, we recommend calling
#> rstan_options(auto_write = TRUE)
#> For within-chain threading using `reduce_sum()` or `map_rect()` Stan functions,
#> change `threads_per_chain` option:
#> rstan_options(threads_per_chain = 1)
#> Do not specify '-march=native' in 'LOCAL_CPPFLAGS' or a Makevars file
rstan_options(threads_per_chain = 4)
stan_model(model_code =model_c)
#> Error in compileCode(f, code, language = language, verbose = verbose): C:/Users/Reece/Documents/R/win-library/4.0/RcppEigen/include/Eigen/src/Core/ProductEvaluators.h:35:90: required from 'Eigen::internal::evaluator<Eigen::Product<Lhs, Rhs, Option> >::evaluator(const XprType&) [with Lhs = Eigen::Product<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<double>, const Eigen::Matrix<double, 1, -1> >, const Eigen::Transpose<Eigen::Matrix<double, -1, 1> > >, Eigen::Matrix<double, -1, -1>, 0>; Rhs = Eigen::Matrix<double, -1, 1>; int Options = 0; Eigen::internal::evaluator<Eigen::Product<Lhs, Rhs, Option> >::XprType = Eigen::Product<Eigen::Product<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<double>, const Eigen::Matrix<double, 1, -1> >, const Eigen::Transpose<Eigen::Matrix<double, -1, 1> > >, Eigen::Matrix<double, -1, -1>, 0>, Eigen::Matrix<double, -1, 1>, 0>]'C:/Users/Reece/Documents/R/win-library/4.0/RcppEigen/include/Eigen/src/Core/Product.h:132:22: required from 'Eigen::internal::dense_product_base<Lhs, Rhs, Option, 6>::operator const Scalar() const [with Lhs = Eigen::Product<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<double>, const Eigen::Matrix<double, 1, -1> >, const Eigen::Transpose<Eigen::Matrix<double, -1, 1> > >, Eigen::Matrix<double, -1, -1>, 0>; Rhs = Eigen::Matrix<double, -1, 1>; int Option = 0; Eigen::internal::dense_product_base<Lhs, Rhs, Option, 6>::Scalar = double]'C:/Users/Reece/Documents/R/win-library/4.0/StanHeaders/include/src/stan/mcmc/hmc/hamiltonians/dense_e_metric.hpp:22:56: required from 'double stan::mcmc::dense_e_metric<Model, BaseRNG>::T(stan::mcmc::dense_e_point&) [with Model = model1c201a45ce__namespace::model1c201a45ce_; BaseRNG = boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> >]'C:/Users/Reece/Documents/R/win-library/4.0/StanHeaders/include/src/stan/mcmc/hmc/hamiltonians/dense_e_metric.hpp:21:10: required from hereC:/Users/Reece/Documents/R/win-library/4.0/RcppEigen/include/Eigen/src/Core/DenseCoeffsBase.h:55:30: warning: ignoring attributes on template argument 'Eigen::internal::packet_traits<double>::type' {aka '__vector(2) double'} [-Wignored-attributes]make: *** [C:/PROGRA~1/R/R-40~1.3/etc/x64/Makeconf:229: file1c20622020cc.o] Error 1
#> Error in sink(type = "output"): invalid connection
Created on 2021-01-27 by the reprex package (v1.0.0)
This is just the basic reduce_sum example from the Stan reference page, but it gives the same error as my own model. I can look into that tomorrow.
This runs fine for me on Linux. I haven't tested it on Windows though, but it's clearly a TBB/env issue. It'd be great if you can continue testing just in case we can isolate it to your specific env or fix it for Windows, if needed.
I can also replicate the issue on Windows with RStan 2.26, so it looks like some Windows-related
Actually, this fails for me even without the call to rstan_options
:
model_c <-"functions {
real partial_sum(int[] y_slice,
int start, int end,
vector x,
vector beta) {
return bernoulli_logit_lpmf(y_slice | beta[1] + beta[2] * x[start:end]);
}
}
data {
int N;
int y[N];
vector[N] x;
}
parameters {
vector[2] beta;
}
model {
int grainsize = 1;
beta ~ std_normal();
target += reduce_sum(partial_sum, y,
grainsize,
x, beta);
}"
library(rstan)
#> Loading required package: StanHeaders
#> rstan (Version 2.26.0.9000, GitRev: 2e1f913d3ca3)
#> For execution on a local, multicore CPU with excess RAM we recommend calling
#> options(mc.cores = parallel::detectCores()).
#> To avoid recompilation of unchanged Stan programs, we recommend calling
#> rstan_options(auto_write = TRUE)
#> Do not specify '-march=native' in 'LOCAL_CPPFLAGS' or a Makevars file
stan_model(model_code =model_c)
#> Error in compileCode(f, code, language = language, verbose = verbose): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ tbb::blocked_range<std::size_t>(0, num_terms, grainsize), worker, ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ partitioner); ~~~~~~~~~~~~ make: *** [C:/Users/he184239/R/R-40~1.3/etc/x64/Makeconf:229: file4a2878a3d2b.o] Error 1
#> Error in sink(type = "output"): invalid connection
Created on 2021-01-28 by the reprex package (v0.3.0)
My system info:
> readLines("~/.R/Makevars.win")
[1] "CXX14FLAGS += -mtune=native -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2"
[2] "CXX14FLAGS += -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-unused-function"
[3] "CXX14FLAGS += -Wno-maybe-uninitialized -Wno-parentheses -Wno-sign-compare -Wno-attributes"
[4] "CXX14FLAGS += -Wno-unused-but-set-variable -Wno-reorder"
> readLines("~/.Renviron")
[1] "R_LIBS_USER = \"C:/Users/he184239/R/win-library/4.0\""
[2] "HOME = \"C:/Users/he184239/Desktop/WD\""
[3] "MAKEFLAGS = \"-j8\""
[4] "PATH=\"${RTOOLS40_HOME}\\usr\\bin;${RTOOLS40_HOME}\\mingw64\\bin;${PATH}\""
[5] "PATH=\"${RTOOLS40_HOME}\\usr\\bin;${RTOOLS40_HOME}\\mingw64\\bin;${PATH}\""
> devtools::session_info("rstan")
- Session info ----------------------------------------------------------------------------------------
setting value
version R version 4.0.3 (2020-10-10)
os Windows 10 x64
system x86_64, mingw32
ui RStudio
language (EN)
collate English_Australia.1252
ctype English_Australia.1252
tz Australia/Perth
date 2021-01-28
- Packages --------------------------------------------------------------------------------------------
! package * version date lib source
assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.3)
backports 1.2.1 2020-12-09 [1] CRAN (R 4.0.3)
BH 1.75.0-0 2021-01-11 [1] CRAN (R 4.0.3)
brio 1.1.1 2021-01-20 [1] CRAN (R 4.0.3)
callr 3.5.1 2020-10-13 [1] CRAN (R 4.0.3)
checkmate 2.0.0 2020-02-06 [1] CRAN (R 4.0.3)
cli 2.2.0 2020-11-20 [1] CRAN (R 4.0.3)
colorspace 2.0-0 2020-11-11 [1] CRAN (R 4.0.3)
crayon 1.3.4 2017-09-16 [1] CRAN (R 4.0.3)
curl 4.3 2019-12-02 [1] CRAN (R 4.0.3)
desc 1.2.0 2018-05-01 [1] CRAN (R 4.0.3)
diffobj 0.3.3 2021-01-07 [1] CRAN (R 4.0.3)
digest 0.6.27 2020-10-24 [1] CRAN (R 4.0.3)
ellipsis 0.3.1 2020-05-15 [1] CRAN (R 4.0.3)
evaluate 0.14 2019-05-28 [1] CRAN (R 4.0.3)
fansi 0.4.2 2021-01-15 [1] CRAN (R 4.0.3)
farver 2.0.3 2020-01-16 [1] CRAN (R 4.0.3)
ggplot2 3.3.3 2020-12-30 [1] CRAN (R 4.0.3)
glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.3)
gridExtra 2.3 2017-09-09 [1] CRAN (R 4.0.3)
gtable 0.3.0 2019-03-25 [1] CRAN (R 4.0.3)
inline 0.3.17 2020-12-01 [1] CRAN (R 4.0.3)
isoband 0.2.3 2020-12-01 [1] CRAN (R 4.0.3)
jsonlite 1.7.2 2020-12-09 [1] CRAN (R 4.0.3)
labeling 0.4.2 2020-10-20 [1] CRAN (R 4.0.3)
lattice 0.20-41 2020-04-02 [2] CRAN (R 4.0.3)
lifecycle 0.2.0 2020-03-06 [1] CRAN (R 4.0.3)
loo 2.4.1 2020-12-09 [1] CRAN (R 4.0.3)
magrittr 2.0.1 2020-11-17 [1] CRAN (R 4.0.3)
MASS 7.3-53 2020-09-09 [2] CRAN (R 4.0.3)
Matrix 1.3-2 2021-01-06 [2] CRAN (R 4.0.3)
matrixStats 0.57.0 2020-09-25 [1] CRAN (R 4.0.3)
mgcv 1.8-33 2020-08-27 [2] CRAN (R 4.0.3)
munsell 0.5.0 2018-06-12 [1] CRAN (R 4.0.3)
nlme 3.1-151 2020-12-10 [2] CRAN (R 4.0.3)
pillar 1.4.7 2020-11-20 [1] CRAN (R 4.0.3)
pkgbuild 1.2.0 2020-12-15 [1] CRAN (R 4.0.3)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.3)
pkgload 1.1.0 2020-05-29 [1] CRAN (R 4.0.3)
praise 1.0.0 2015-08-11 [1] CRAN (R 4.0.3)
prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.0.3)
processx 3.4.5 2020-11-30 [1] CRAN (R 4.0.3)
ps 1.5.0 2020-12-05 [1] CRAN (R 4.0.3)
R6 2.5.0 2020-10-28 [1] CRAN (R 4.0.3)
RColorBrewer 1.1-2 2014-12-07 [1] CRAN (R 4.0.3)
Rcpp 1.0.6 2021-01-15 [1] CRAN (R 4.0.3)
RcppEigen 0.3.3.9.1 2020-12-17 [1] CRAN (R 4.0.3)
D RcppParallel 5.0.2 2020-06-24 [1] CRAN (R 4.0.3)
rematch2 2.1.2 2020-05-01 [1] CRAN (R 4.0.3)
rlang 0.4.10 2020-12-30 [1] CRAN (R 4.0.3)
rprojroot 2.0.2 2020-11-15 [1] CRAN (R 4.0.3)
rstan 2.26.0.9000 2021-01-20 [1] local
rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.0.3)
scales 1.1.1 2020-05-11 [1] CRAN (R 4.0.3)
StanHeaders 2.26.0.9000 2021-01-20 [1] local
testthat 3.0.1 2020-12-17 [1] CRAN (R 4.0.3)
tibble 3.0.5 2021-01-15 [1] CRAN (R 4.0.3)
utf8 1.1.4 2018-05-24 [1] CRAN (R 4.0.3)
V8 3.4.0 2020-11-04 [1] CRAN (R 4.0.3)
vctrs 0.3.6 2020-12-17 [1] CRAN (R 4.0.3)
viridisLite 0.3.0 2018-02-01 [1] CRAN (R 4.0.3)
waldo 0.2.3 2020-11-09 [1] CRAN (R 4.0.3)
withr 2.4.1 2021-01-26 [1] CRAN (R 4.0.3)
[1] C:/Users/he184239/R/win-library/4.0
[2] C:/Users/he184239/R/R-4.0.3/library
D -- DLL MD5 mismatch, broken installation.
@andrjohns Thanks for the confirmation! It looks like a TBB issue on Window, likely a compatibility with the version of TBB shipped in RcppParallel
, which may or may not affect cmdstanr
depending on which TBB version it links to. Can you test this with cmdstanr
?
I'm also interested in testing an external oneTBB installation, which is supported by both RcppParallel
and Stan math
now with environment variables.
No problems with cmdstanr
0.2.2 using cmdstan
2.26
Ok. Thanks! I'll look into it.
@andrjohns @Reece844 Test with my development version of RcppParallel
:
remove.packages("RcppParallel")
remotes::install_github("hsbadr/RcppParallel@develop")
What version of TBB does RcppParallel ship? I believe we did not upgrade TBB in Stan Math because we wanted to stay compatible with RcppParallel.
What version of TBB does RcppParallel ship? I believe we did not upgrade TBB in Stan Math because we wanted to stay compatible with RcppParallel.
A 2-years older version that also have a conflicting tbb/parallel_reduce.h
! See https://github.com/hsbadr/RcppParallel/commit/26cbda93dc037de90f14d4d015adba48ce396476. I'll submit this to RcppParallel
after testing.
Also, a dirty workaround that works is to replace tbb/parallel_reduce.h
with the one in Stan math
.
This might be the best place to leave a link to this discrepancy obtained using the following:
> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7
Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib
locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8
attached base packages:
[1] stats graphics grDevices datasets utils methods base
other attached packages:
[1] cmdstanr_0.3.0 rstan_2.26.0.9000 StanHeaders_2.26.0.9000
loaded via a namespace (and not attached):
[1] tidyselect_1.1.0 xfun_0.20 purrr_0.3.4 V8_3.4.0
[5] colorspace_2.0-0 vctrs_0.3.6 generics_0.1.0 stats4_4.0.3
[9] loo_2.4.1 rlang_0.4.10 pkgbuild_1.2.0 pillar_1.4.7
[13] glue_1.4.2 DBI_1.1.1 matrixStats_0.57.0 lifecycle_0.2.0
[17] posterior_0.1.3 munsell_0.5.0 gtable_0.3.0 codetools_0.2-16
[21] inline_0.3.17 knitr_1.31 callr_3.5.1 ps_1.5.0
[25] curl_4.3 parallel_4.0.3 fansi_0.4.2 Rcpp_1.0.6
[29] renv_0.12.0 scales_1.1.1 backports_1.2.1 checkmate_2.0.0
[33] RcppParallel_5.0.2 jsonlite_1.7.2 abind_1.4-5 gridExtra_2.3
[37] ggplot2_3.3.3 processx_3.4.5 dplyr_1.0.3 grid_4.0.3
[41] cli_2.2.0 tools_4.0.3 magrittr_2.0.1 tibble_3.0.5
[45] crayon_1.3.4 pkgconfig_2.0.3 ellipsis_0.3.1 data.table_1.13.6
[49] prettyunits_1.1.1 assertthat_0.2.1 R6_2.5.0 compiler_4.0.3
A 2-years older version that also have a conflicting
tbb/parallel_reduce.h
! See hsbadr/RcppParallel@26cbda9. I'll submit this toRcppParallel
after testing.
Sorry for the delay, I can confirm that both of these fix the compilation problems on my machine when using the reduce_sum.
Sorry for the delay, I can confirm that both of these fix the compilation problems on my machine when using the reduce_sum.
Great! Thanks for testing. I've submitted a PR to RcppParallel
: https://github.com/RcppCore/RcppParallel/pull/151
It looks like GPU support will also need an additional header file included: see #715
The file already exists in the Math library, but isn't carried across when StanHeaders
is built
@SteveBronder was there anything else (that you can remember) it needed?
@bgoodri @rok-cesnovar Now that #887 has been tested, could you merge it so that new features can be submitted in separate PRs that can be easily reviewed?
It looks like GPU support will also need an additional header file included: see #715
@andrjohns OpenCL headers have been added in StanHeaders
by https://github.com/stan-dev/rstan/pull/887/commits/72f6aff630b2f4f26987df62c4b30e1c17a7381c. You can now #include <CL/opencl.h>
or #include <CL/cl2.hpp>
. You can check the packages that will be uploaded as artifacts here.
Thanks @hsbadr, that seems to have mostly done it. It finds the file fine, but there's a new error during linking:
Error in dyn.load(libLFile) :
unable to load shared object '/tmp/RtmpjVHjjp/file140e7192cfb.so':
/tmp/RtmpjVHjjp/file140e7192cfb.so: undefined symbol: clReleaseContext
There are no issues compiling with GPU enabled via cmdstanr on the same system.
@rok-cesnovar Have you seen that before? It looks OpenCL related. For reference, my Makevars
file is:
CXX14FLAGS += -O3 -std=c++1y -march=native -mtune=native -Wno-attributes
CXX14FLAGS += -Wno-ignored-attributes -Wno-deprecated-declarations
CXX14FLAGS += -lOpenCL -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0
CXX14FLAGS += -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120
CXX14FLAGS += -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS
https://github.com/stan-dev/rstan/pull/887 is huge and I think the PR thread will eventually become impossible to follow. So I am opening this issue where you can post anything that is missing in the alpha 2.26 version.
Thanks @hsbadr for working on this massive pull request!
I think the checklist for making this happen is as follows:
[ ] Check that it works for the early-adopters/devs on all OS-es for R4.x (R-release)
[ ] Check that it works for the early-adopters/devs on all OS-es for R3.x (R-oldrel)
I think after this we can merge the PR and deal with smaller issues later.
[ ] Make a beta-call on Discourse for other users with a disclaimer that it might break other packages that depend on rstan.
[ ] Hash out issues after the beta call
[ ] Try to get this to CRAN - this is where my knowledge stops. I expect this is where the "fun" will start.