Milestones and features

jayrobwilliams commented 5 years ago

@jkarreth had the very good point that we should start developing milestones and deciding what features/issues should be associated with which milestones. I think the starting point is what do we want to see in a feature complete, version 1.0.0?

I think the biggest/most consequential question we need to answer first is do we consider MCMC diagnostics to be part of Bayesian post estimation, or is that an auxiliary task we can leave to other packages? coda and ggmcmc implement lots of diagnostics and diagnostic plots in addition to presentation of results with coefficient/caterpillar plots and density plots. Do we want to try and include this diagnostic functionality in BayesPostEst, or do we want to focus on presenting results as we've defined them so far?

Decided below to focus on substantive post estimation.

jkarreth commented 5 years ago

Hmm, I would vote against MCMC diagnostics. As you write, this is already pretty well done in coda and ggmcmc (and some in rstan & bayesplot, or even superdiag), so I'd see a lot of potential for duplicating. If either of you feels different though, please post below!

jkarreth commented 5 years ago

Here are some things I suggest for the next versions; please feel free, all, to edit as you see fit!

0.1.0

[x] any JOSS review requests

0.2.0

[x] #41 add coefficient plot function
[x] #21 add marginal effects plot

0.3.0

[ ] #43 automatically extract posterior samples for all functions

later versions

[ ] #18 expand probability functions to other GLMs
[ ] think about probability function expansions to multilevel models, if at all possible

ShanaScogin commented 5 years ago

We had some requests at ICPSR to add more functionality to mcmcTab() - any thoughts on what might make that better? I also agree on leaving out mcmc diagnostics, and I wholeheartedly support expanding to other glms.

jayrobwilliams commented 5 years ago

I agree that we should leave MCMC diagnostics out since there are very mature packages that handle most/all of them.

Other features I've considered:

A ridge plot function like mcmcFDplot()but just for the posterior distributions of parameters
A caterpillar plot function since lots of people do latent variable models in Bayes
Expanding #21 add marginal effects plot to also handle a binary/categorical interaction

If these sound good, I can work them into the milestones in @jkarreth's post above.

jkarreth commented 5 years ago

Thanks for these suggestions!

A ridge plot function like mcmcFDplot() but just for the posterior distributions of parameters
- I like the idea; could it be an option to the coefplot function? E.g. ridge = TRUE or something like that. Especially for ridge plots, we'd want to add a warning that parameters should be roughly on the same scale; the most frequent thing I've seen in homework assignments is widely different scales, which render these plots much less useful.
A caterpillar plot function since lots of people do latent variable models in Bayes
- I like that too. But can we think about how to streamline this vs. just taking the output and plotting it "by hand"? That can be already pretty quick, so let's think about what might be able to be improved.
Expanding #21 add marginal effects plot to also handle a binary/categorical interaction
- Great idea. I have some code for a frequentist version here that does that automatically (for binary interactions at least).

Lastly, before we expand too much, I just came across the very cool bayestestR package. It doesn't duplicate what we do but may already cover some things that we may have planned.

jkarreth commented 5 years ago

@ShanaScogin do you want to submit the current version (post-JOSS) to CRAN as version 0.1.0? I adjusted the version milestones above.

It may depend a bit on what @jayrobwilliams would like to do with the coefficient plot and marginal effects functions. If we add these soon, we could wait and put 0.2.0 on CRAN.

ShanaScogin commented 5 years ago

I'm totally up for submitting this version to CRAN. Rob, what do you think?

jayrobwilliams commented 5 years ago

I think we should submit as is, since mcmcReg() and the better parameter extraction in mcmcTab() are pretty big improvements. I think it makes sense to target the other new functions for 0.2.0, but there are also lots of small enhancements I can apply from the beginning of mcmcReg() that will take a while to add to all the other functions.

jkarreth commented 5 years ago

Great! Let me make a couple small changes to the vignette and I will post here again when it's ready for your review/submitting to CRAN.

jkarreth commented 5 years ago

@ShanaScogin vignette should be good to go - just added a reference and a few line breaks in the code.

ShanaScogin commented 5 years ago

Great - am submitting now. Will let everyone know when it's up. I'll pull Andy's request after that and start working on the enhancement in issue #50

jayrobwilliams commented 5 years ago

mcmcCoefPlot and mcmcMargEff have been merged into develop and should be good to go!

ShanaScogin commented 5 years ago

Great! Cran release for v0.1.0 went out Sunday as well. Look forward to working on the new stuff for the next couple months.

jayrobwilliams commented 5 years ago

@jkarreth your comment on caterpillar plots

I like that too. But can we think about how to streamline this vs. just taking the output and plotting it "by hand"? That can be already pretty quick, so let's think about what might be able to be improved.

is prescient since today I realized that caterpillar plots are just our existing mcmcCoefPlot() with the parameters sorted by their means/medians. I think it makes sense to add a sort argument to the function, which would easily allow users to create caterpillar plots and simplify code maintenance since we won't have two separate functions with 90% code in common.

Also, I agree that we should just add a ridge argument as well like you suggested above. I'll implement that and add a warning about standardizing inputs if, let's say, the distance between the smallest and largest point is more than 5 times the average uncertainty interval?

jkarreth commented 5 years ago

👍 on the sort arg for mcmcCoefPlot()!

I'd be happy with your suggestion about ridge.

Lastly, what are your thoughts about the following for mcmcMargEff:

add a way to also handle a binary/categorical interaction
incorporating some of the features of the interflex package, esp. the histogram and L/M/H box plots
add a second plot underneath with the posterior probability of the estimate being </> 0 (as in my teaching example)

jayrobwilliams commented 5 years ago

Those all sound good to me. I think the binary/categorial interaction and sorting the point estimates are good targets for 0.2.0, and the interflex and Pr > or < 0 plots make sense for a later version since they're more involved. Once I get those additions implemented and @ShanaScogin gets the static docs site working do we want to release 0.2.0?

ShanaScogin / BayesPostEst