mjskay
commented
6 months ago Cool, yeah. If I understand the intention, it wouldn't use rvar's specialized indexing with [ and [[, but could probably implement many of the other functions of rvars?
This could be implemented as a codification of what draws_of(<rvar>) returns. Would have to modify it a bit because rvar keeps the nchains and weights` attributes on the surrounding object rather than the array it contains, but in principle I think that could be adjusted.
I'm not sure what to call such a type --- I think it wouldn't be a "draws" type because the other "draws" types are collections of variables rather than a single multidimensional variable, so you wouldn't be able to convert any "draws" object to one of this new type (just as you can't convert other "draws" types to just a single rvar). Maybe this is a "raw_rvar" or a "rvar_array" or something? Or "variable_array"? Hmm.
This also raises another thought I've been mulling over, which is I wonder if it would be helpful to have some variations on rvar with different backing formats. E.g. one solution for #234 is to have a variant of rvar that is a list-column format. This would be less efficient for some operations (like matrix multiplication), but would allow storage in data.tables and may be more efficient for some other operations too (like storage in tibbles).
This perhaps suggests two families of formats within {posterior}: formats that represent collections of variables (the "draws" formats), and formats that represent single multidimensional variables (currently just "rvar", but maybe this family needs a name if we expand it).
paul-buerkner
The discussion about a data.table format reminded me of something I wanted to bring up for a while. I wonder if we can have a non-rvar format that can handle high dimensional arrays of posterior draws? Otherwise behaving like a draws_matrix, i.e., storing chains only as an attribute. If I remember correctly, this is kind of how we store draws in rvars behind the scenes without making this its own format.
The reason for bringing this up is that some brms post-processing function return multi-dimensional output (+ the draws dim) and we don't really represent this in posterior so far except for in rvar. Using rvar as a default output in brms however would be too much a backwards compatibility breaking change (although it will become a non-default option in brms 3.0) and perhaps a bit too adventurous too. That is why I am looking for a "low-level" alternative that resembles what brms already does, namely just outputting high-dimensional arrays.
What are your thoughts on this? Specifically, what does @mjskay think?