Open petermacp opened 10 months ago
Yeah, we've had to do similar things on large numbers of predictions. It probably would be helpful to have some infrastructure to make this kind of workflow easier, e.g. by providing columns to use to define batches to generate predictions from or something like that.
Re: ndraws specifically, can you make a predictions from one row of this model with all draws fine? I guess I'm not clear on exactly what the problem is...
Thanks so much.
I guess my question is whether you have any tips to make the add_epred_draws() call as fast as possible, and output a smaller object?
Peter
You could try using add_epred_rvars instead, as the rvar format in a data frame will be more compact than the format output by add_epred_draws.
Beyond that the obvious way to speed this up is to run the computations for different splits in parallel.
I am exploring working out a workflow for using
add_epred_draws()
and subsequent calculations when the number of potential predictors is very large. Appreciate the answer might be "just use a smaller prediction matrix" 🤣, but interested to see what is possible.Following along with this example from Andrew Heiss, I have constructed a
brms
regression model, that fits well:We make a very large prediction matrix, comprising 5 million rows, with all combinations of predictors (just for this example).
Of course, when we try to
add_epred_draws()
usingnd3_matrix
in thenewdata=
argument, we very quickly run out of memory.So instead, I wondered if it would be possible split
nd3_matrix
into more manageable chunks, and write to a database to allow more efficient post-processing, like this:This seems to work (within a reasonable hour or so), and allows me to work with predictions using
dbplyr
that might not otherwise be possible. But wondering if there are any ways to future optimise this, and potentially increase thendraws
possible?Thanks!