search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
721 stars 38 forks source link

[REVIEW]: bayestestR: Describing Effects and their Uncertainty, Existence and Significance within the Bayesian Framework #1541

Closed whedon closed 5 years ago

whedon commented 5 years ago

Submitting author: @DominiqueMakowski (Dominique Makowski) Repository: https://github.com/easystats/bayestestR Version: 0.2.5 Editor: @cMadan Reviewer: @paul-buerkner, @tjmahr Archive: 10.5281/zenodo.3361605

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/1d180e6004a0dd1e6b235eb24fe66276"><img src="http://joss.theoj.org/papers/1d180e6004a0dd1e6b235eb24fe66276/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/1d180e6004a0dd1e6b235eb24fe66276/status.svg)](http://joss.theoj.org/papers/1d180e6004a0dd1e6b235eb24fe66276)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@paul-buerkner & @tjmahr, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @cMadan know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @paul-buerkner

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @tjmahr

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 5 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @paul-buerkner, @tjmahr it looks like you're currently assigned to review this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

paul-buerkner commented 5 years ago

I have just finished my review and have very few minor comments.

cMadan commented 5 years ago

Thank you for the thorough review, @paul-buerkner!

DominiqueMakowski commented 5 years ago

Dear @paul-buerkner, thanks a lot for your comments! We addressed them in this PR:

Reviewer 1 (@paul-buerkner)

  • [x] All the citations in the software paper should list a DOI (if they have one) as per reviewer checklist above.

  • Added DOIs for all refs but the following (none was found):

    • see package (here)
    • rstanarm package
    • BayesFactor package
    • Mill's "Objective Bayesian Precise Hypothesis Testing"
    • Multiple Comparisons with BayesFactor, Part 2 (Morey's blog, 2015)
    • Practical bayesian optimization of machine learning algorithms (Snoek's proceedings, 2012)
    • Jeffrey's Theory of Probability book
  • [x] The definition of the maximum a-posteriori value as "the most probable value" is not entirely correct for continuous parameters. Instead it is the value with the highest density (which still have probability zero for continuous parameters).
  • We changed its definition to the following:

"find the Highest Maximum A Posteriori (MAP) estimate of a posterior, i.e., the value associated with the highest probability density (the "peak" of the posterior distribution). In other words, it is an estimation of the mode for continuous parameters."

  • [x] you seem to use the abbreviation MAP both for the maximum a posteriori and the maximum a-priori value. This will likely confuse readers.
  • This was likely an error and was addressed by replacing instances of the latter by the former (the maximum a posteriori).

Please note that there are still some references for which we did not find a DOI: we continue our search in the meantime. We hope you will find the revised version satisfying ☺️

paul-buerkner commented 5 years ago

Looks good to me.

cMadan commented 5 years ago

@tjmahr, are you still able to review this submission?

@paul-buerkner, thanks again!!

tjmahr commented 5 years ago

I would like to review it but won't be able to look at it until next week.

cMadan commented 5 years ago

@tjmahr, no problem, thanks for following up!

strengejacke commented 5 years ago

@whedon generate pdf

whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

BarryDeCicco commented 5 years ago

Please take me off of the list of reviewers. I have received over a dozen e-mails on four projects today. The first project, yesterday, generated ~30 e-mails to me, most of them not understandable.IMHO, your process for reviewing needs serious work. Sincerely, Barry On Thursday, July 25, 2019, 6:38:25 AM EDT, whedon notifications@github.com wrote:

👉 Check article proof 📄 👈

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

cMadan commented 5 years ago

@BarryDeCicco, you should unsubscribe from the GitHub notifications for this repository, or otherwise change your notification settings. See the second post (https://github.com/openjournals/joss-reviews/issues/1541#issuecomment-507707086) in this review thread (same information is available in every review thread).

tjmahr commented 5 years ago

I worked with version 0.2.4, the most recent on the GitHub repository, although the version mentioned in the checklist is 0.2.3

Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

I do not see anything for contribution guidelines. I would add a CONTRIBUTING.md file.

Comments

The amount of documentation to support the package is very generous. For my review, however, I focused on the README and the software paper.

This package borrows a lot from the Kruschke school of Bayesian inference. HDI and ROPE are a distinct feature of his tutorials and textbook; one does not see them very often in works by, say, Stan developers. Therefore, this package is tremendously useful for those reading his tutorials, or for people like me, who occasionally will quantify an effect with a ROPE percentage or who want to learn about using Bayes factors.

The ROPE procedure and other indices use the highest density interval. Is there any option to use an equal-tailed interval?

README

I was confused by the README. When I see R code followed immediately by a plot, I assume that the R code produced the plot. But the functions in the README produce text output (which is not included in the README) and they do not produce plots. I would include the text output of the R code. I would also note that the figures there are diagrams meant to illustrate the statistical concept. The software paper does a good job of making this point clear.

I don't see a demo for eti().

Moreover, 89 is the highest prime number that does not exceed the already unstable 95% threshold (McElreath, 2015).

The primeness of 89 is not important. McElreath's choice of 89 in Statistical Rethinking text was to illustrate that interval widths are arbitrary and that there is nothing special about 95 or 90 compared to 89.

equivalence_test() a Test for Practical Equivalence based on the

Needs a verb.

I don't understand the Bayes Factor diagram in the README.

a range of -0.05 to -0.05.

This range is the same number twice.

Savage-Dickey density ratio is computed

Should this have a reference?

Probability of a Value

density_at() isn't doing computing a probability. I would remove estimate_probability() and probability_at() because they are just aliases for density functions and density is the more appropriate term.

I don't see a demo for the area under the curve functions.

Documentation

A ROPE-based p of 97% means that there is a probability of .97 that a parameter (described by its posterior distribution) is outside the ROPE. On the contrary, a ROPE-based p of -97% means that there is a probability of .97 that the parameter is inside the ROPE. (R/p_rope.R)

I don't understand how a p-value can get a negative percentage. What would a 0% p-value mean? If this index doesn't act like a familiar p-value, it is probably the wrong name for it.

Software paper

The first mention of bayestestR in the second paragraph is awkward. Specifically, the text shifts from talking about common ways to describe effects in a Bayesian framework to talking about the features of the package:

Additionally, bayestestR also focuses on implementing a Bayesian null-hypothesis testing framework ...

It's great that the output of point_estimate() prints out mean/median/map to make it clear what value is being used.

However, bayestestR functions also include plotting capabilities via the see package (Lüdecke, Waggoner, Ben-Shachar, & Makowski, 2019).

I don't see any plotting examples in the README or documentation pages. I see plotting methods in the NAMESPACE.

I think it would worthwhile to demonstrate that the functions demoed in the article also work on models. For example, I can call p_direction() and bayesfactor_parameters() directly on a model and get the results for each parameter. One of the key contributions of this package is that it can make these indices immediately available to users who are comfortable with brms and rstanarm.

Proofreading concerns

Every reference of Kruschke spells out the author's full name.

Figure 2 should be referenced in the text.

(i.e. the difference

Needs a comma.

The Bayesian framework allows to neatly delineate

Allows one.

developped

Nevertheless, in the absence of user-provided values, bayestestR will automatically find an appropriate range

Nevertheless doesn't make sense.

bases on prior and posterior samples

Based.

The system for building the references section should protect some words from being converted to lowercase. (In LaTeX, this is done with {}). Right now, for example, it says Brms: An r package for bayesian multilevel models using stan but I would make sure that the system produces brms: An R package for Bayesian multilevel models using Stan.

DominiqueMakowski commented 5 years ago

Dear @tjmahr, thanks a lot for your thorough review. We have addressed them in this PR:

Features

  • [x] The ROPE procedure and other indices use the highest density interval. Is there any option to use an equal-tailed interval?

We added a ci_method argument in rope() to allow for ETI to be used.

  • [x] density_at() isn't doing computing a probability. I would remove estimate_probability() and probability_at() because they are just aliases for density functions and density is the more appropriate term.

We removed the two aliases with probability. We also clarified in the documentation that it is pertaining to the value of the density function.

README

  • [x] I do not see anything for contribution guidelines. I would add a CONTRIBUTING.md file.

We added a contributing file.

  • [x] I would include the text output of the R code [in the README].

We have additionally included the text output from the R Code in the README.

  • [x] I would also note that the figures there are diagrams meant to illustrate the statistical concept.

We have added a sentence to point out that these figures are meant to illustrate the statistical concepts, and pointed the readers to the see-package, where plotting-methods are provided:

"The following figures are meant to illustrate the (statistical) concepts behind the functions. However, for most functions, plot()-methods are available from the see-package."

  • [x] I don't see a demo for eti().

We have added a demo for eti() to the README.

  • [x] "Moreover, 89 is the highest prime number that does not exceed the already unstable 95% threshold (McElreath, 2015)." The primeness of 89 is not important. McElreath's choice of 89 in Statistical Rethinking text was to illustrate that interval widths are arbitrary and that there is nothing special about 95 or 90 compared to 89.

We have rephrased the sentence to emphasize the idea behind choosing the 89 as CI-level:

"Moreover, 89 indicates the arbitrariness of interval limits - its only remarkable property is being the highest prime number that does not exceed the already unstable 95% threshold (McElreath, 2015)"

Furthermore, although already implied in the paper, we also emphasized the point of arbitrariness on the paper as well.

  • [x] _"equivalencetest() a Test for Practical Equivalence based on the" Needs a verb.

We added a verb to the sentence:

equivalence_test() is a Test for Practical Equivalence based on the...

  • [x] I don't understand the Bayes Factor diagram in the README.

We have added a paragraph to explain the figure more in detail:

The lollipops represent the density of a point-null on the prior distribution (the blue lollipop on the dotted distribution) and on the posterior distribution (the red lollipop on the yellow distribution). The ratio between the two - the Svage-Dickey ratio - indicates the degree by which the mass of the parameter distribution has shifted away from or closer to the null.

  • [x] "a range of -0.05 to -0.05." This range is the same number twice.

Thanks, we fixed the typo!

  • [x] "Savage-Dickey density ratio is computed" Should this have a reference?

Thanks, we have added a reference, and furthermore added a reference-list to the end of the README.

  • [x] I don't see a demo for the area under the curve functions.

See comment from TJ below, no longer necessary.

Documentation

  • [x] ROPE-based p documentation: I don't understand how a p-value can get a negative percentage. What would a 0% p-value mean? If this index doesn't act like a familiar p-value, it is probably the wrong name for it.

We have clarified the documentation of this index and underlined its exploratory nature. We also made clear that the negative sign reflects the direction of the index (wether in corresponds to significance or non-significance), rather than actual negative probabilities, which indeed make no sense.

The ROPE-based \emph{p}-value is an exploratory and non-validated index representing the 
maximum percentage of \link[=hdi]{HDI} that does not contain (or is entirely contained, in which 
case the value is prefixed with a negative sign), in the negligible values space defined by the 
\link[=rope]{ROPE}. It differs from the ROPE percentage, \emph{i.e.}, from the proportion of a given
 CI in the ROPE, as it represents the maximum CI values needed to reach a ROPE proportion of 0\% 
or 100\%. Whether the index reflects the ROPE reaching 0\% or 100\% is indicated through the sign:
 a negative sign is added to indicate that the probability corresponds to the probability of a not 
significant effect (a percentage in ROPE of 100\%). For instance, a ROPE-based \emph{p} of 97\% 
means that there is a probability of .97 that a parameter (described by its posterior distribution) is 
outside the ROPE. In other words, the 97\% HDI is the maximum HDI level for which the percentage 
in ROPE is 0\%. On the contrary, a ROPE-based p of -97\% indicates that there is a probability of .97 
that the parameter is inside the ROPE (percentage in ROPE of 100\%). A value close to 0\% would 
indicate that the mode of the distribution falls perfectly at the edge of the ROPE, in which case the 
percentage of HDI needed to be on either side of the ROPE becomes infinitely small. Negative values 
do not refer to negative values \emph{per se}, simply indicating that the value corresponds to non-
significance rather than significance.

Paper

  • [x] The first mention of bayestestR in the second paragraph is awkward. Specifically, the text shifts from talking about common ways to describe effects in a Bayesian framework to talking about the features of the package: "Additionally, bayestestR also focuses on implementing a Bayesian null-hypothesis testing framework ..."
  • [x] "However, bayestestR functions also include plotting capabilities via the see package (Lüdecke, Waggoner, Ben-Shachar, & Makowski, 2019).": I don't see any plotting examples in the README or documentation pages. I see plotting methods in the NAMESPACE.
  • [x] I think it would worthwhile to demonstrate that the functions demoed in the article also work on models. For example, I can call p_direction() and bayesfactor_parameters() directly on a model and get the results for each parameter.

Proofreading

  • [x] Every reference of Kruschke spells out the author's full name.

Hopefully fixed (changed the name in the .bib file). However, I am not sure why would that happen. One possible reason is disambiguation, yet all instances were written the same way...

  • [x] Figure 2 should be referenced in the text.
  • [x] "(i.e. the difference": Needs a comma.
  • [x] "The Bayesian framework allows to neatly delineate": Allows one.
  • [x] developped
  • [x] Nevertheless, in the absence of user-provided values, bayestestR will automatically find an appropriate range: Nevertheless doesn't make sense.
  • [x] bases on prior and posterior samples: Based
  • [x] The system for building the references section should protect some words from being converted to lowercase. (In LaTeX, this is done with {}). Right now, for example, it says Brms: An r package for bayesian multilevel models using stan but I would make sure that the system produces brms: An R package for Bayesian multilevel models using Stan.

Typos have been fixed.

We hope you will be satisfied with the revisions ☺️

strengejacke commented 5 years ago

@whedon generate pdf

whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

tjmahr commented 5 years ago

We have added a paragraph to explain the figure more in detail:

The lollipops represent the density of a point-null on the prior distribution (the blue lollipop on the dotted distribution) and on the posterior distribution (the red lollipop on the yellow distribution). The ratio between the two - the Svage-Dickey ratio - indicates the degree by which the mass of the parameter distribution has shifted away from or closer to the null.

Just fix the typo in Savage-Dickey, and I'm satisfied.

tjmahr commented 5 years ago

Also, thanks, in particular, for adding the ETI functionality for the ROPE methods.

DominiqueMakowski commented 5 years ago

typo has been fixed!

cMadan commented 5 years ago

Thanks for the thorough review, @tjmahr!

@DominiqueMakowski, it looks like there are no outstanding issues, is that correct?

DominiqueMakowski commented 5 years ago

@cMadan That is correct ☺️

cMadan commented 5 years ago

@DominiqueMakowski, great!!

To move forward with accepting your submission, there are a few last things to take care of:

You may find this helpful: https://guides.github.com/activities/citable-code/

DominiqueMakowski commented 5 years ago

@cMadan we have tagged a v0.2.5 release and a Zenodo identifier ☺️

DOI

cMadan commented 5 years ago

@whedon set 0.2.5 as version

whedon commented 5 years ago

OK. 0.2.5 is the version.

cMadan commented 5 years ago

@whedon set 10.5281/zenodo.3361605 as archive

whedon commented 5 years ago

OK. 10.5281/zenodo.3361605 is the archive.

cMadan commented 5 years ago

@DominiqueMakowski, perfect, thank you!

@openjournals/joss-eics, I think we're all set to accept here!

danielskatz commented 5 years ago

@whedon generate pdf

whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
danielskatz commented 5 years ago

Thanks to @paul-buerkner and @tjmahr for reviewing and @cMadan for editing

whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

danielskatz commented 5 years ago

👋 @DominiqueMakowski - please see https://github.com/easystats/bayestestR/pull/217 and merge it - also carefully check the rest of the bib to make sure I didn't miss anything else (e.g., words in lower case that should be in upper case, odd periods at the end of titles, etc.)

strengejacke commented 5 years ago

@danielskatz Thanks for the thorough reading! I have read the paper and checked all hyperlinks, everything looks good so far.

I'll go through the paper and check the references now.

strengejacke commented 5 years ago

@whedon generate pdf

whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

danielskatz commented 5 years ago

In addition, here are some changes for the paper. (in a PR that I forgot to add but has now been merged) :)

strengejacke commented 5 years ago

Thanks for the language editing! i have gone through the references and found some minor changes. I will hand over to @DominiqueMakowski for the final check.

danielskatz commented 5 years ago

Ok - please let me know when you & @DominiqueMakowski are done, then we can proceed.

DominiqueMakowski commented 5 years ago

@whedon generate pdf

whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

DominiqueMakowski commented 5 years ago

@danielskatz Thanks a lot for your changes! @cMadan I think we are good to go ☺️

danielskatz commented 5 years ago

@whedon accept

whedon commented 5 years ago 
Attempting dry run of processing paper acceptance...