Closed wlandau closed 3 years ago
On reflection, I do not think this is worth a special feature. Users can load the relevant output into memory and specify the existing rds
or qs
format.
self$draws()
try(self$sampler_diagnostics(), silent = TRUE)
try(self$init(), silent = TRUE)
Related: @mike-lawrence and others, do you think there would be value in a tarchetypes
-like package for cmdstan
models? Some possible target archetypes:
tar_cmdstan_fit("model.stan")
: Fit a Stan model, call the methods in https://github.com/wlandau/targets/issues/211#issuecomment-724022390, and save the object in qs
format.tar_cmdstan_fit_summary("model.stan")
: Same, except the return value is a lightweight data frame of posterior summaries and diagnostics.tar_cmdstan_validate("model.stan", sim_function, batches = 20, sims_per_batch = 40)
: define several targets to repeatedly simulate from a model and determine how often the true parameters in the data are recaptured in credible intervals.That sounds for sure useful!
This weekend, I implemented something like https://github.com/wlandau/targets/issues/211#issuecomment-724070231 in a new package called stantargets
. It's not public yet, but it is fully fleshed out and working. I will post a link as soon as I get permission to share.
stantargets
is now open source! It is an extension to targets
for Stan-powered Bayesian data analysis. stantargets
makes it super easy to set up useful pipelines without having to write many functions or think about branching. It supports all the features of cmdstanr
(MCMC, variational Bayes, optimization), and it supports both single-fit workflows and multi-rep simulation studies (vignettes here).
Update: I am trying to continue what I started with stantargets
and cultivate a whole ecosystem of these packages: https://wlandau.github.io/targetopia.html. My background is in Bayesian statistics, so that's where I am starting. But I also want to branch out to machine learning. @mattwarkentin, do you think the stuff people do with torch
has enough workflow patterns to materialize in a stantargets
-like package?
Hey @wlandau. Upon seeing targetopia
, I immediately thought about whether I have anything worthwhile to contribute. My mind went to torch
, but my hesitation was based on the fact that I really have yet to use or inspect stantargets
et al. so I don't really know well enough what those packages do to reduce the friction of using targets
. I'll try to dig into stantargets
this weekend and see if it inspires any thoughts about how something similar could be warranted for torch
.
I immediately thought about whether I have anything worthwhile to contribute.
I think you do. You know targets
super well, and you are experienced with torch
, which I have not used in a real-world project.
My mind went to torch, but my hesitation was based on the fact that I really have yet to use or inspect stantargets et al. so I don't really know well enough what those packages do to reduce the friction of using targets.
Targetopia packages aim to automatically build in options and techniques that would either be annoying to implement manually or too advanced for the majority of users. Examples:
stantargets::tar_stan_mcmc()
creates a target for the output object and targets for each of the friendly summaries the user might want.tarchetypes::tar_rep()
is the simplest example of this. Dynamic branching is too complicated and intimidating for most users, and even for me it can be a headache to manually set up a batching scheme. stantargets
functions like tar_stan_mcmc_rep_summary()
use this same idea.targets
to know they need format = "file"
in tar_target()
to track files.memory = "transient"
and/or garbage_collection = TRUE
for that target. tar_stan_mcmc_rep_draws()
does this.deployment
(and maybe storage
and memory
for that matter). stantargets
needs to be able to run on remote workers without access to the Stan model file. So stantargets
works around this with a local target that reads the lines of the file and uses deployment = "main"
(see the compile
argument of various functions). Then, subsequent targets automatically accept those file lines in memory, write them to a temporary file, and compile the model locally (one compilation per batch).At some point I plan to write general guidance for targetopia developers.
Thanks for the detailed response, Will. This is very helpful for understanding how these add-on packages contribute to the targetopia
. I will mull it over a bit more and try to sketch out some torch
workflow patterns that I think might lend themselves well to this ecosystem.
I am glad you are interested. With some good torch
workflow patterns, I think a new torch
-powered targetopia package is possible. cc @skeydan and @dfalbel.
That looks really nice! I's be happy to help! Let me know if you need any development in the torch side too.
Thanks, glad to hear it! The torch
vignettes are super nice, and they are helping me begin to wrap my head around usage patterns. If anything else strikes you as a particularly ubiquitous archetype, please let us know.
@mattwarkentin, FYI: the R Targetopia now has its own website with a more detailed contributing guide.
Prework
Proposal
We could consider accommodating
CmdStanFit
objects withtar_target(format = "cmdstanr")
. Related: https://github.com/stan-dev/cmdstanr/issues/340.