I noticed that you're computing the gradient by manually going through the data points. After looking back at my README, I realized this was done by another contributor. I think he was trying to save memory for the intermediate nodes. Not sure what your thoughts are about this, but I think if a user can load an nxp matrix of data, we can afford to allocate O(n) extra memory for the autodiff. In that case, we can do something simpler (untested code):
auto expr = ad::bind(ad::norm(y-ad::dot(X, theta)));
(I really regret naming it norm when it's really "norm-squared"...). This should only allocate 2*(2n + 1) extra doubles in total for the intermediate nodes.
I think we can also pretty easily generalize the dimension of theta. We can just take p = theta_hat.size() for example.
const auto p = theta_hat.size();
Eigen::VectorXd theta_adj(p);
theta_adj.setZero(); // Set adjoints to zeros.
// Initialize variable.
ad::VarView<double, ad::vec> theta(theta_hat.data(), theta_adj.data(), p);
// Create expr. Use a row buffer to store data. Then we only need to manipulate data when
// looping.
auto expr = ad::bind(ad::norm(y - ad::dot(X, theta)));
// Differentiate and get loss
auto loss = ad::autodiff(expr);
Thanks again for making this!!
Just had a few things about the linear regression example.
I noticed that you're computing the gradient by manually going through the data points. After looking back at my README, I realized this was done by another contributor. I think he was trying to save memory for the intermediate nodes. Not sure what your thoughts are about this, but I think if a user can load an nxp matrix of data, we can afford to allocate O(n) extra memory for the autodiff. In that case, we can do something simpler (untested code):
(I really regret naming it
norm
when it's really "norm-squared"...). This should only allocate 2*(2n + 1) extra doubles in total for the intermediate nodes.I think we can also pretty easily generalize the dimension of theta. We can just take
p = theta_hat.size()
for example.Maybe a simpler implementation to replace: https://github.com/eddelbuettel/rcppfastad/blob/dbcbcd17073f620de7cb646b5df42a853d6f2de6/src/linear_regression.cpp#L23-L49 with