search

stan-dev / math

The Stan Math Library is a C++ template library for automatic differentiation of any order using forward, reverse, and mixed modes. It includes a range of built-in functions for probabilistic modeling, linear algebra, and equation solving.
https://mc-stan.org
BSD 3-Clause "New" or "Revised" License
733 stars 184 forks source link

autodiff for matrix add and subtract #1787

Open bob-carpenter opened 4 years ago

bob-carpenter commented 4 years ago

Description

There are lots of functions that don't have reverse mode specializations.

Matrix operations

All of these would be very useful, espcially the basic arithmetic of the first five and inverse_spd because it's so important.

All of the constraints would be good to specialize. These get used everywhere (the least important are corr_constrain and cov_matrix_constrain---the cholesky forms are more important for our recommended usages).

Then there are some reductions:

There are probably more.

Feel free to solve one or more of these in a PR.

Current Version:

v3.1.0

betanalpha commented 4 years ago

While specializing it would also be useful to consider direct adjoint-Jacobian transpose implementations which can provide significant memory savings for matrix operations.

On Mar 20, 2020, at 6:26 PM, Bob Carpenter notifications@github.com wrote:

Description

There are lots of functions that don't have reverse mode specializations.

Matrix operations

All of these would be very useful, espcially the basic arithmetic of the first five and inverse_spd because it's so important.

add

subtract

minus

elt_multiply

elt_divide

diag_post_multiply

diag_pre_multiply

quad_form_diag

chol2inv

`inverse_spd

All of the constraints would be good to specialize. These get used everywhere (the least important are corr_constrain and cov_matrix_constrain---the cholesky forms are more important for our recommended usages).

lb_constrain ub_constrain lub_constrain corr_constrain cov_matrix_constrain cholesky_factor_constrain cholesky_corr_constrain offset_multiplier_constrain Then there are some reductions:

mean sd variance There are probably more.

Feel free to solve one or more of these in a PR.

Current Version:

v3.1.0

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/stan-dev/math/issues/1787, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALU3FRVL45AGSDJ2BHEXR3RIPURPANCNFSM4LQVVS2Q.

wds15 commented 4 years ago

Sounds like there is a lot of performance on the table to grab here! Looking forward to those benchmarks.

bob-carpenter commented 4 years ago

Definitely direct adjoint-Jacobian implementations are the way to go to avoid all the virtual function calls and collapse the autodiff graph.

dpsimpson commented 4 years ago

If no one minds, I'll have a shot at some of these over the weekend.

bob-carpenter commented 4 years ago

@dpsimpson: All yours.