search

leeper / margins

An R Port of Stata's 'margins' Command
https://cloud.r-project.org/package=margins
Other
260 stars 40 forks source link

Unit Specific Variances Possible Speed-Up #109

Open EdJeeOnGitHub opened 5 years ago

EdJeeOnGitHub commented 5 years ago

Do unit specific variances in build_margins.R strictly need to be calculated row-by-row?

I've been calculating SEs in blocks of 10,000 manually since the default implementation was taking too long - my case might be unique as I already know the gradient function for every row but if it isn't I'm happy to try and speed this section up.

Apologies if I'm missing something and this is the only way to deal with calculating the SEs.

leeper commented 5 years ago

There should definitely be ways to improve this significantly. I intended to improve but then discovered most people don't seem to need these, so I turned it off by default (to improve overall speed) and haven't received a lot of requests for improvement. If you have ideas for how to update, I'm open to suggestions and PRs.

EdJeeOnGitHub commented 5 years ago

Currently I'm calculating the Jacobian symbolically and then just applying Jacobian %*% vcov %*% Jacobian' in blocks of 10,000 and taking the square root of the diagonal.

I know we can't use the symbolic differentiation in margins but I imagine just finding the Jacobian row by row and then using the vectorised/block matrix method will offer a speed up over the current method.

If this all sounds sensible to you I'll give it a go.

leeper commented 5 years ago

Go for it!

EdJeeOnGitHub commented 5 years ago

Hi, I forked the repo and implemented the method above but I couldn't get any significant speed-ups - it looks like calculating the Jacobian row-by-row is by far the slowest part of unit_ses = TRUE so vectorising the the Jacobian sandwich separately afterwards doesn't help much.

If I can think of any way to speed up the Jacobian calculation with unit-specific standard errors I'll give it another shot but for now I don't think this is going anywhere.

leeper commented 5 years ago

Bummer. Definitely let me know if you come up with something better. I'll leave this open.