Closed whedon closed 3 years ago
Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @bkamins, @kescobo it looks like you're currently assigned to review this paper :tada:.
:star: Important :star:
If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/JuliaCon/proceedings-review) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿
To fix this do the following two things:
For a list of things I can do to help you, just type:
@whedon commands
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
@whedon generate pdf
Attempting PDF compilation. Reticulating splines etc...
Just an update: I am putting together example code to generate all the figures and the table in the paper. I will post this on the repo soon. I've add a license. I have added a table and made some other changes to the paper, and I am still working on editing the paper - I will upload a revised version soon. I will also add the conference presentation and any related code to the repo once I have that together.
Thank you for all this. So probably it is best if we wait for the update before reviewing. Right? Please let us know when you are done. Thank you.
Some minor points while you update the text are (this is based on just skimming over the paper - I will read it in derail after the final version is uploaded):
begin
-end
and for
code blocksYes, waiting for the update before reviewing makes sense. I will incorporate all your suggestions above, thanks for these!
Alright - I'm back. Looks like a lot has happened, but we're waiting on some updates before moving forward with review. Is that right?
Yep, working on my presentation right now! Will update this after the presentation.
Some small code review comments:
Array{Real}
can be probably replaced by Array{Float64}
and the code shall run faster.wt_draws
twice but do not define wp_draws
below in the code and probably you do not need to call Array
when you extract values from ct
(I can check it if we have reproducible codes)
Thanks Bogumil! I had spotted the wt_draws error but not the Real. Also, thanks for answering Stefan K's question more clearly than I did at my presentation - I chatted with him later and he said you had explained it to him.
At the airport in Baltimore waiting for my flight right now. Wish I could have stayed for the hackathon. Was a fantastic conference. Hope to chat with you at JuliaCon 2020!
@tszanalytics - thank you for a very nice presentation (and I did not think that you knew that it was me who made the remark - but it turns out you did 😄).
I am obsessed with the difference in interpretation of hypothesis test results in frequentist and Bayesian inference (and actually I think that Bayesian interpretation is more intuitive for most people) and your work fits perfectly into this picture.
@bkamins - Actually I didn't know it was you at the time, I only found out when I chatted with Stefan later. I wish I had known so I could have chatted with you. We must get together at the next time. My flight delayed so more time to work on the proceedings revisions 😄
@whedon generate pdf
Attempting PDF compilation. Reticulating splines etc...
@bkamins Ok, recompiled the pdf here. Thanks for pointing me in the right direction.
@tszanalytics
I think your work is an interesting case study. Here are my comments to your submission (please treat them as suggestions and decide what you think should be incorporated).
rng_example.jl
:
using Random
on top of the fileusing StatsPlots
on top of the file# savefig("approaching_certainty_03.png")
should be removed from the file as it is commented outlogistic_trajectory_by_week_example.jl
https://github.com/tszanalytics/BayesTesting.jl
is not a registered package; can you please add instructions how it should be loaded by the user (in particular which commit should be used as a reference); also please place all using
statements at the top of the file (now they are in several places and there are duplicates)juliacon_proceedings2019_code.jl
logistic_trajectory_by_week_example.jl
A major comment is to give more discussion of your BayesTesting.jl package (design goals, functionality etc. - see questions for reviewers to see what kind of information is expected). Of course please try to keep in the page limit of the proceedings when doing this (this is a soft suggestion).
@tszanalytics did you finish the corrections from @bkamins ?
I'm a bit confused about whether I'm supposed to be evaluating just the code in the Proceedings repo, BayesTesting.jl
or both. This review assumes both, please let me know if I'm mistaken.
This manuscript presents a justification for and examples of using Bayesian inference in biomedical research, specifically outlining a method using julia to generate "Bayesian p-values" as a bridge from the current frequentist methods typically employed in biological contexts. As a biologist that's just starting to learn about Bayesian methods, I'm excited to have this resource and to have a force in the julia community pushing towards more robust statistical methods.
The manuscript could benefit from some minor updates to present a consistent voice and to provide additional explanation / context in the form of citations for a number of claims. The accompanying code should have substantially more documentation / clarity, both in terms of content and presentation. That said, this effort is definitely worth publishing with some modification.
Comments on the manuscript can be found here: 10.21105.jcon.00036_KSB_comments.pdf
I largely agree with @bkamins ' comments here regarding the need for additional documentation (in a perfect world, there would be runable jupyter notebooks, or you could use something like Literate.jl
) and tests.
DataDeps
annotation for it so it's easily acquirable to reproduce your analyses.Indeed that's both, thanks
On Wed, Aug 21, 2019, 00:46 Kevin Bonham notifications@github.com wrote:
I'm a bit confused about whether I'm supposed to be evaluating just the code in the Proceedings repo, BayesTesting.jl or both. This review assumes both, please let me know if I'm mistaken. Overall
This manuscript presents a justification for and examples of using Bayesian inference in biomedical research, specifically outlining a method using julia to generate "Bayesian p-values" as a bridge from the current frequentist methods typically employed in biological contexts. As a biologist that's just starting to learn about Bayesian methods, I'm excited to have this resource and to have a force in the julia community pushing towards more robust statistical methods.
The manuscript could benefit from some minor updates to present a consistent voice and to provide additional explanation / context in the form of citations for a number of claims. The accompanying code should have substantially more documentation / clarity, both in terms of content and presentation. That said, this effort is definitely worth publishing with some modification. Manuscript
Comments on the manuscript can be found here: 10.21105.jcon.00036_KSB_comments.pdf https://github.com/JuliaCon/proceedings-review/files/3522868/10.21105.jcon.00036_KSB_comments.pdf Code
I largely agree with @bkamins https://github.com/bkamins ' comments here regarding the need for additional documentation (in a perfect world, there would be runable jupyter notebooks, or you could use something like Literate.jl) and tests. Other Minor Comments
- There's a "Third Author" in the authors list. I'm guessing that's part of the template slipping through?
- tszanalytics/BayesTesting.jl#11 https://github.com/tszanalytics/BayesTesting.jl/issues/11
- Is the raw data for the analyses you perform in the paper accessible? Would be great to have a DataDeps annotation for it so it's easily acquirable to reproduce your analyses.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/JuliaCon/proceedings-review/issues/36?email_source=notifications&email_token=AB2FDMRSMWRC57T27UWC5EDQFRX35A5CNFSM4IDVDKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4X4ITI#issuecomment-523224141, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2FDMRWLRO7ELAPKC4O53TQFRX35ANCNFSM4IDVDKJQ .
Thanks for the comments all. I have been a bit busy the last week or so, but now this is top of my list. I will get to work revising the paper and the code, etc. to address the comments. Adding a jupyter notebook sounds like a good idea also, so I will have a go at putting one together.
I've just got back to working on this and realized there is some confusion about what is being reviewed. The presentation and the proceedings submission for JuliaCon 2019 was not about the package BayesTesting.jl - I gave a presentation about that in 2018 - so I don't believe that is part of the work currently under review. However, I would be happy to receive any comments and suggestions with regard to the package. I haven't had time to work on developing it further in some time, but hope to do so soon and plan on registering it very soon.
ping @tszanalytics, did you have time to work on the corrections?
@whedon generate pdf
Attempting PDF compilation. Reticulating splines etc...
@whedon generate pdf
Attempting PDF compilation. Reticulating splines etc...
@whedon generate pdf
Attempting PDF compilation. Reticulating splines etc...
PDF failed to compile for issue #36 with the following error:
Berry2017' on page 1 undefined on input line 33 Citation
Berry20ll' on page 1 undefined on input line 30
Citation Mills2018' on page 1 undefined on input line 24 Citation
Mills2018' on page 2 undefined on input line 73
Citation Mills2019' on page 1 undefined on input line 22 Citation
Mills2019' on page 1 undefined on input line 24
Citation Mills2019' on page 2 undefined on input line 73 Citation
Mills2019' on page 2 undefined on input line 76
Citation Mills2019' on page 2 undefined on input line 78 Citation
Ramos2010' on page 2 undefined on input line 58
Citation Strawn2017' on page 1 undefined on input line 24 Citation
Strawn2018' on page 1 undefined on input line 24
Citation Strawn2018' on page 2 undefined on input line 78 Citation
Strawn2018a' on page 1 undefined on input line 24
Citation Strawn2019' on page 1 undefined on input line 24 Citation
Wass2019' on page 1 undefined on input line 33
Citation bezanson2017julia' on page 1 undefined on input line 20 Reference
eq:hoh1' on page 2 undefined on input line 60
Reference eq:hoh1' on page 2 undefined on input line 82 Reference
eq:odds' on page 2 undefined on input line 82
Reference eq:pval' on page 2 undefined on input line 82 Reference
fig:pars_diff' on page 2 undefined on input line 82
Latexmk: Summary of warnings:
Latex failed to resolve 5 reference(s)
Latex failed to resolve 19 citation(s)
Rule 'pdflatex': File changes, etc:
Changed files, or newly in use since previous run(s):
'paper.aux'Berry2017' on page 1 undefined on input line 33 Citation
Berry20ll' on page 1 undefined on input line 30
Citation Mills2018' on page 1 undefined on input line 24 Citation
Mills2018' on page 2 undefined on input line 73
Citation Mills2019' on page 1 undefined on input line 22 Citation
Mills2019' on page 1 undefined on input line 24
Citation Mills2019' on page 2 undefined on input line 73 Citation
Mills2019' on page 2 undefined on input line 76
Citation Mills2019' on page 2 undefined on input line 78 Citation
Ramos2010' on page 2 undefined on input line 58
Citation Strawn2017' on page 1 undefined on input line 24 Citation
Strawn2018' on page 1 undefined on input line 24
Citation Strawn2018' on page 2 undefined on input line 78 Citation
Strawn2018a' on page 1 undefined on input line 24
Citation Strawn2019' on page 1 undefined on input line 24 Citation
Wass2019' on page 1 undefined on input line 33
Citation bezanson2017julia' on page 1 undefined on input line 20 Reference
eq:odds' on page 2 undefined on input line 82
Reference eq:pval' on page 2 undefined on input line 82 Reference
fig:pars_diff' on page 2 undefined on input line 82
Latexmk: Summary of warnings:
Latex failed to resolve 3 reference(s)
Latex failed to resolve 19 citation(s)
Rule 'pdflatex': File changes, etc:
Changed files, or newly in use since previous run(s):
'paper.out'Latexmk: List of undefined refs and citations:
Citation Berry2017' on page 1 undefined on input line 33 Citation
Berry20ll' on page 1 undefined on input line 30
Citation Mills2018' on page 1 undefined on input line 24 Citation
Mills2018' on page 2 undefined on input line 73
Citation Mills2019' on page 1 undefined on input line 22 Citation
Mills2019' on page 1 undefined on input line 24
Citation Mills2019' on page 2 undefined on input line 73 Citation
Mills2019' on page 2 undefined on input line 76
Citation Mills2019' on page 2 undefined on input line 78 Citation
Ramos2010' on page 2 undefined on input line 58
Citation Strawn2017' on page 1 undefined on input line 24 Citation
Strawn2018' on page 1 undefined on input line 24
Citation Strawn2018' on page 2 undefined on input line 78 Citation
Strawn2018a' on page 1 undefined on input line 24
Citation Strawn2019' on page 1 undefined on input line 24 Citation
Wass2019' on page 1 undefined on input line 33
Citation bezanson2017julia' on page 1 undefined on input line 20 Reference
eq:odds' on page 2 undefined on input line 82
Reference eq:pval' on page 2 undefined on input line 82 Reference
fig:pars_diff' on page 2 undefined on input line 82
Latexmk: Summary of warnings:
Latex failed to resolve 3 reference(s)
Latex failed to resolve 19 citation(s)
Failure to make 'paper.pdf'
Collected error summary (may duplicate other messages):
pdflatex: Command for 'pdflatex' gave return code 1
Refer to 'paper.log' for details
Looks like we failed to compile the PDF
@whedon generate pdf
Attempting PDF compilation. Reticulating splines etc...
PDF failed to compile for issue #36 with the following error:
Berry2017' on page 1 undefined on input line 33 Citation
Berry20ll' on page 1 undefined on input line 30
Citation Mills2018' on page 1 undefined on input line 24 Citation
Mills2018' on page 2 undefined on input line 73
Citation Mills2019' on page 1 undefined on input line 22 Citation
Mills2019' on page 1 undefined on input line 24
Citation Mills2019' on page 2 undefined on input line 73 Citation
Mills2019' on page 2 undefined on input line 76
Citation Mills2019' on page 2 undefined on input line 78 Citation
Ramos2010' on page 2 undefined on input line 58
Citation Strawn2017' on page 1 undefined on input line 24 Citation
Strawn2018' on page 1 undefined on input line 24
Citation Strawn2018' on page 2 undefined on input line 78 Citation
Strawn2018a' on page 1 undefined on input line 24
Citation Strawn2019' on page 1 undefined on input line 24 Citation
Wass2019' on page 1 undefined on input line 33
Citation bezanson2017julia' on page 1 undefined on input line 20 Reference
eq:hoh1' on page 2 undefined on input line 60
Reference eq:hoh1' on page 2 undefined on input line 82 Reference
eq:odds' on page 2 undefined on input line 82
Reference eq:pval' on page 2 undefined on input line 82 Reference
fig:pars_diff' on page 2 undefined on input line 82
Latexmk: Summary of warnings:
Latex failed to resolve 5 reference(s)
Latex failed to resolve 19 citation(s)
Rule 'pdflatex': File changes, etc:
Changed files, or newly in use since previous run(s):
'paper.aux'Berry2017' on page 1 undefined on input line 33 Citation
Berry20ll' on page 1 undefined on input line 30
Citation Mills2018' on page 1 undefined on input line 24 Citation
Mills2018' on page 2 undefined on input line 73
Citation Mills2019' on page 1 undefined on input line 22 Citation
Mills2019' on page 1 undefined on input line 24
Citation Mills2019' on page 2 undefined on input line 73 Citation
Mills2019' on page 2 undefined on input line 76
Citation Mills2019' on page 2 undefined on input line 78 Citation
Ramos2010' on page 2 undefined on input line 58
Citation Strawn2017' on page 1 undefined on input line 24 Citation
Strawn2018' on page 1 undefined on input line 24
Citation Strawn2018' on page 2 undefined on input line 78 Citation
Strawn2018a' on page 1 undefined on input line 24
Citation Strawn2019' on page 1 undefined on input line 24 Citation
Wass2019' on page 1 undefined on input line 33
Citation bezanson2017julia' on page 1 undefined on input line 20 Reference
eq:odds' on page 2 undefined on input line 82
Reference eq:pval' on page 2 undefined on input line 82 Reference
fig:pars_diff' on page 2 undefined on input line 82
Latexmk: Summary of warnings:
Latex failed to resolve 3 reference(s)
Latex failed to resolve 19 citation(s)
Rule 'pdflatex': File changes, etc:
Changed files, or newly in use since previous run(s):
'paper.out'Latexmk: List of undefined refs and citations:
Citation Berry2017' on page 1 undefined on input line 33 Citation
Berry20ll' on page 1 undefined on input line 30
Citation Mills2018' on page 1 undefined on input line 24 Citation
Mills2018' on page 2 undefined on input line 73
Citation Mills2019' on page 1 undefined on input line 22 Citation
Mills2019' on page 1 undefined on input line 24
Citation Mills2019' on page 2 undefined on input line 73 Citation
Mills2019' on page 2 undefined on input line 76
Citation Mills2019' on page 2 undefined on input line 78 Citation
Ramos2010' on page 2 undefined on input line 58
Citation Strawn2017' on page 1 undefined on input line 24 Citation
Strawn2018' on page 1 undefined on input line 24
Citation Strawn2018' on page 2 undefined on input line 78 Citation
Strawn2018a' on page 1 undefined on input line 24
Citation Strawn2019' on page 1 undefined on input line 24 Citation
Wass2019' on page 1 undefined on input line 33
Citation bezanson2017julia' on page 1 undefined on input line 20 Reference
eq:odds' on page 2 undefined on input line 82
Reference eq:pval' on page 2 undefined on input line 82 Reference
fig:pars_diff' on page 2 undefined on input line 82
Latexmk: Summary of warnings:
Latex failed to resolve 3 reference(s)
Latex failed to resolve 19 citation(s)
Failure to make 'paper.pdf'
Collected error summary (may duplicate other messages):
pdflatex: Command for 'pdflatex' gave return code 1
Refer to 'paper.log' for details
Looks like we failed to compile the PDF
@matbesancon Not sure what is going on now. The pdf file looks good when I compile it with TeXstudio, but things are moved around when I compile it here. Now it won't compile at all. Somebody strangle whedon for me.
I've updated the .jl files, added two jupyter notebooks, and substantially revised the paper in light of the comments (thanks everyone for all the great suggestions, etc.). I'll will post a more detailed response once whedon decides to generate the pdf!
try compiling locally using latexmk
, which is what is used on the server
@whedon generate pdf
Attempting PDF compilation. Reticulating splines etc...
@whedon generate pdf
Attempting PDF compilation. Reticulating splines etc...
@matbesancon @bkamins @kescobo
Thanks everyone for all the great comments and suggestions. I think the paper is greatly improved as a result. I believe everything on the github repo is now up to date and all the changes are incorporated there. Here's a summary of many of the revisions in response to the reviewer suggestions:
• Changed title. • Added email addresses. • Put citations before periods and commas. • Fixed typos in equations (3) and (4) and added cite to Gelman (2004). • Tried changing from Real to Float64, but for this model it turns out it has to be Real because of use of auto-differentation. • Reworded 1st sentence in intro. to make it clearer. • Reworded 3rd paragraph so it makes more sense and to remove “clean code”. • Provided a citation to support the “iron law” and “institutional inertia” statements. • Provided a citation for the Behrens-Fisher problem. • Edited references section to correct several typos, etc. • Fixed typos in the figures. • Increased the font size for the figures to the extent possible (I found this to be particularly difficult!). • Reduced the size of Fig. 3 by only including two sets of time plot + density plot, and explained its purpose further. This allowed the addition of a new figure (Fig. 5) that provides an example of posterior density plots for treatment, placebo and difference. • Reduced the length of the paper so that it fits on 5 pages.
Please let me know if I've missed anything and I will endeavor to fix it.
Thanks, Jeff
Thanks for edits - I'll take a look first thing next week.
@tszanalytics Thanks for the edits. I am largely satisfied with the changes. There are just a few outstanding questions/concerns.
Project.toml
and Manifest.toml
for (just one that includes all the packages necessary for both should be fine). This will be useful for future readers so they can get the exact versions of the packages you use. Also would be good to include the julia version number you're using I think.For the second point, if the manuscript is about and uses BayesTesting.jl, it should be reviewed with it. Otherwise if the software is contained in the repo of the paper, it should be as suggested versioned & documented
I'm not sure about the citations. It appears to put them in alphabetical order by first author name. I am using \bibliographystyle{juliacon} \bibliography{ref} as the bibliography style as per instructions, so that must be what is wanted for the proceedings?
My presentation and this paper is not about the package and only uses the package somewhat tangentially. I presented a preliminary version of the package at JuliaCon 2018, and plan on updating and registering it to provide a more complete package by next year, so I don't think the package itself is ready for evaluation. The paper and presentation is about application of Julia to analysis of RCTs and meta analysis.
Unfortunately I must admit that I don't really understand toml files and their usage as yet. I will have to read up on this. I believe I would have to specify a particular project to enable generation of toml files that contain only the packages for that project? I'm not sure how to do this yet.
I included the Julia version and computer specifics in the paper on page 3 (at end of right column).
Thanks!
Ad. 3 - this is very easy:
activate .
add WHATEVER_PACKAGES_YOU_REQUIRE_FOR_THE_CODES_TO_WORK
In the directory of your project you will have the files @kescobo mentions.
Submitting author: !--author-handle-->@tszanalytics<!--end-author-handle-- (Jeffrey A Mills) Repository: https://github.com/tszanalytics/Juliacon2019 Branch with paper.md (empty if default branch): Version: Editor: !--editor-->@matbesancon<!--end-editor-- Reviewers: @bkamins, @kescobo Archive: Pending
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@bkamins & @kescobo, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://proceedings.juliacon.org/guide/reviewers. Any questions/concerns please let @matbesancon know.
✨ Please try and complete your review in the next two weeks ✨
Review checklist for @bkamins
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Paper format
paper.tex
file include a list of authors with their affiliations?Content
Review checklist for @kescobo
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek supportPaper format
paper.tex
file include a list of authors with their affiliations?Content