expose_stan_functions fails on functions postfixed with `_lp`

[bgoodri]

Summary:

expose_stan_functions does not work with functions that modify target, which worked previously

Description:

If you try, there is a compiler error

Reproducible Steps:

expose_stan_functions this

functions {
  void foo_lp() {
    target += normal_lpdf(0.5 | 0, 1);
  }
}

Current Output:

In file included from file7d8f62dde969.cpp:1:
In file included from /usr/local/lib/R/site-library/rstan/include/exporter.h:1:
In file included from /usr/local/lib/R/site-library/Rcpp/include/RcppCommon.h:168:
In file included from /usr/local/lib/R/site-library/Rcpp/include/Rcpp/as.h:25:
/usr/local/lib/R/site-library/Rcpp/include/Rcpp/internal/Exporter.h:31:28: error: no matching constructor for initialization of 'stan::math::accumulator<double>'
                    Exporter( SEXP x ) : t(x){}
                                         ^ ~
/usr/local/lib/R/site-library/Rcpp/include/Rcpp/as.h:87:41: note: in instantiation of member function 'Rcpp::traits::Exporter<stan::math::accumulator<double> >::Exporter' requested here
            ::Rcpp::traits::Exporter<T> exporter(x);
                                        ^
/usr/local/lib/R/site-library/Rcpp/include/Rcpp/as.h:152:26: note: in instantiation of function template specialization 'Rcpp::internal::as<stan::math::accumulator<double> >' requested here
        return internal::as<T>(x, typename traits::r_type_traits<T>::r_category());
                         ^
/usr/local/lib/R/site-library/Rcpp/include/Rcpp/InputParameter.h:46:49: note: in instantiation of function template specialization 'Rcpp::as<stan::math::accumulator<double> >' requested here
        ReferenceInputParameter(SEXP x_) : obj( as<T>(x_) ){}
                                                ^
file7d8f62dde969.cpp:75:77: note: in instantiation of member function 'Rcpp::ReferenceInputParameter<stan::math::accumulator<double> >::ReferenceInputParameter' requested here
    Rcpp::traits::input_parameter< stan::math::accumulator<double>& >::type lp_accum__(lp_accum__SEXP);
                                                                            ^
/usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat/fun/accumulator.hpp:25:7: note: candidate constructor (the implicit copy constructor) not viable: cannot convert argument of incomplete type 'SEXP' (aka 'SEXPREC *') to 'const stan::math::accumulator<double>' for 1st argument
class accumulator {
      ^
/usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat/fun/accumulator.hpp:33:3: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
  accumulator() : buf_() {}
  ^

and the relevant part of the generated C++ file is

void foo_lp(double& lp__, stan::math::accumulator<double>& lp_accum__, std::ostream* pstream__);
RcppExport SEXP sourceCpp_1_foo_lp(SEXP lp__SEXP, SEXP lp_accum__SEXP, SEXP pstream__SEXP) {
BEGIN_RCPP
    Rcpp::RNGScope rcpp_rngScope_gen;
    Rcpp::traits::input_parameter< double& >::type lp__(lp__SEXP);
    Rcpp::traits::input_parameter< stan::math::accumulator<double>& >::type lp_accum__(lp_accum__SEXP);
    Rcpp::traits::input_parameter< std::ostream* >::type pstream__(pstream__SEXP);
    foo_lp(lp__, lp_accum__, pstream__);
    return R_NilValue;
END_RCPP
}

Expected Output:

None

RStan Version:

develop (2.18) branch from GitHub

R Version:

3.5

Operating System:

Debian

[bgoodri]

But it breaks tests in other R packages and changes the behavior of expose_stan_functions() without being readily apparent to the person calling the function in order to do something that is completely foreign in the R / Python workflow.

On Thu, Aug 23, 2018 at 9:30 PM Krzysztof Sakrejda notifications@github.com wrote:

The current pr works without any more gsub or default values, you'd just have to pass all the requires args Stan requires

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/rstan/issues/550#issuecomment-415623145, or mute the thread https://github.com/notifications/unsubscribe-auth/ADOrquxhVd4W7I7yflstHdHXlG1wCUnNks5uT1cmgaJpZM4WDJD5 .

[sakrejda]

Sure, if somebody is relying on undocumented text munging on c++ code sometimes their code will need updating. They should've used an explicit return to begin with. As I said elsewhere I don't object to the code munging, I just don't want to take responsibility for maintaining the code munging.

[bob-carpenter]

RNGs maintain state, but I agree that it's unusual behavior for R and that we should maintain existing behavior.

What is the existing behavior for _lp functions? Is there a way to recover the target increment or is it just hidden?

[bgoodri]

The existing, and documented, behavior is that the user-defined function ending in _lp is passed (by value) an initial value of lp (which defaults to 0) that gets incremented from there. If users want to chain those together, then they need to return target(); from the first function call and pass the returned value to the lp argument of the next function call.

On Fri, Aug 24, 2018 at 4:56 AM Bob Carpenter notifications@github.com wrote:

RNGs maintain state, but I agree that it's unusual behavior for R and that we should maintain existing behavior.

What is the existing behavior for _lp functions? Is there a way to recover the target increment or is it just hidden?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/rstan/issues/550#issuecomment-415697479, or mute the thread https://github.com/notifications/unsubscribe-auth/ADOrqr4YUILgMzAw3i48Xkszxu1IkP3Pks5uT7-sgaJpZM4WDJD5 .

[bob-carpenter]

I found this doc, but couldn't follow it.

LP functions If a user-defined Stan function ends in _lp, then it can modify the log-probability used by Stan to evaluate Metropolis proposals or as an objective function for optimization. When exposing such functions to R, a lp argument will be added to the formals. This `lpargument defaults to zero, but a double precision scalar may be passed to this argument when the function is called from R. Such a user-defined Stan function can terminate with return target(); or can executeprint(target())`; to verify that the calculation is correct.

I don't see from this how I'd call the exposed function, because I dont know what "formals" are. I see there are "formal arguments" but I don't know what other kind there are (are the ... not formal?).

Is lp__ the name of an argument, as in I'd use lp__ = x to pass in a value?

I take it the target() has to be inside the Stan program? I didn't see doc for it as an RStan function.

And I take it there's no persistence of target() across multiple calls the way there is in a Stan program.

I take it the argument being passed is passed by value (I don't even know if there is pass-by-reference in R) so that it doesn't get modified the way the actual Stan function works?

[bgoodri]

The answer to all of those is "yes". And the formals of foo_lp are the arguments to the function lp. There are no informals.

On Fri, Aug 24, 2018 at 9:18 AM Bob Carpenter notifications@github.com wrote:

I found this doc, but couldn't follow it.

LP functions If a user-defined Stan function ends in _lp, then it can modify the log-probability used by Stan to evaluate Metropolis proposals or as an objective function for optimization. When exposing such functions to R, a lp argument will be added to the formals. This lp argument defaults to zero, but a double precision scalar may be passed to this argument when the function is called from R. Such a user-defined Stan function can terminate with returntarget(); or can execute print(target()); to verify that the calculation is correct.

I don't see from this how I'd call the exposed function, because I dont know what "formals" are. I see there are "formal arguments" but I don't know what other kind there are (are the ... not formal?).

Is lp the name of an argument, as in I'd use lp = x to pass in a value?

I take it the target() has to be inside the Stan program? I didn't see doc for it as an RStan function.

And I take it there's no persistence of target() across multiple calls the way there is in a Stan program.

I take it the argument being passed is passed by value (I don't even know if there is pass-by-reference in R) so that it doesn't get modified the way the actual Stan function works?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/rstan/issues/550#issuecomment-415755941, or mute the thread https://github.com/notifications/unsubscribe-auth/ADOrqkBoUvIA5Kv4RUaQv5CBiy1Fs4n-ks5uT_0TgaJpZM4WDJD5 .