[REVIEW]: Model dispersion with PRISM; an alternative to MCMC for rapid analysis of models

whedon commented 5 years ago

Submitting author: @1313e (Ellert van der Velden) Repository: https://github.com/1313e/PRISM Version: v1.0.0 Editor: @arokem Reviewer: @fonnesbeck Archive: 10.5281/zenodo.2572736

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf"><img src="http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf/status.svg)](http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@fonnesbeck, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @arokem know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @fonnesbeck

Conflict of interest

[x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Version: Does the release version given match the GitHub release (v1.0.0)?
[x] Authorship: Has the submitting author (@1313e) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Authors: Does the paper.md file include a list of authors with their affiliations?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

whedon commented 5 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @fonnesbeck it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

whedon commented 5 years ago

Attempting PDF compilation. Reticulating splines etc...

whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

1313e commented 5 years ago

As mentioned in the blog here (http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing), it must be clear to all parties involved that a JOSS submission is associated with an AAS Journals submission. Given that there is currently (as far as I know) not a defined way of doing so, I would like to mention here that there is an AAS submission associated with this JOSS submission.

@fonnesbeck @arokem Do you need the AAS Journals submission reference for this by any chance?

arfon commented 5 years ago

Given that there is currently (as far as I know) not a defined way of doing so, I would like to mention here that there is an AAS submission associated with this JOSS submission.

@1313e many thanks for letting us know about this - I think you're the first!

@fonnesbeck - when you get a chance, please review this blog post from late last year: http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing . In particular, please let us know if you have any objection to JOSS receiving a small donation from AAS publishing for this.

1313e commented 5 years ago

@1313e many thanks for letting us know about this - I think you're the first!

Yeah, the news about that collaboration came just in time before my paper submission, so I am not surprised I am the first one.

1313e commented 5 years ago

@arfon May I btw mention that AAS Journals has no system in place at all that a submission is being accompanied by a JOSS submission? I simply mentioned it in the extra notes to the editor, but I have not heard anything back yet.

arfon commented 5 years ago

I simply mentioned it in the extra notes to the editor, but I have not heard anything back yet.

Yes, I think this is still a work in progress.

/ cc @crawfordsm who is the software editor for AAS so he has some visibility into this review too.

crawfordsm commented 5 years ago

Apologies, unfortunately, I did not see a note about this on the AAS side. I'll have to follow up about that, but thanks for letting me know! As this is the first paper, it will be great to use as an example of the process.

1313e commented 5 years ago

Apologies, unfortunately, I did not see a note about this on the AAS side. I'll have to follow up about that, but thanks for letting me know! As this is the first paper, it will be great to use as an example of the process.

@crawfordsm Do you, by any chance, need the submission reference, to make it easier to find which paper it actually is that I submitted to AAS Journals?

crawfordsm commented 5 years ago

It's okay, I have the information. I am the Scientific Editor for the paper, so I can monitor it here as well as on the AAS side. I realize now that I had just overlooked the information. As of right now, we do not have a formal process for this, but appreciate your patience as we figure out the details.

arokem commented 5 years ago

Hi @arfon: thanks for the heads up about this. Do I understand correctly that the process here is essentially the same as for other JOSS papers? That is, review and acceptance here are not tied in any way to the AAS paper. Is that correct?

arfon commented 5 years ago

Hi @arfon: thanks for the heads up about this. Do I understand correctly that the process here is essentially the same as for other JOSS papers? That is, review and acceptance here are not tied in any way to the AAS paper. Is that correct?

That's mostly correct. We may want to hold this paper back from final acceptance until we know what the DOI for the AAS paper will be (so we can cite it).

arokem commented 5 years ago

Understood! OK - @fonnesbeck - if this all sounds agreeable to you, please do go ahead with your review. Once the review process here is over, we can circle back and see what we need to do. Thanks!

1313e commented 5 years ago

That's mostly correct. We may want to hold this paper back from final acceptance until we know what the DOI for the AAS paper will be (so we can cite it).

@arfon That would be required anyways, given that I am citing the AAS Journals submission for the results. I felt that putting them into the JOSS paper would be a bit pointless, given that they are already in the AAS submission and it would never fit into 1k words.

I hope that is okay btw, as I couldn't find any guidelines on this.

arfon commented 5 years ago

I felt that putting them into the JOSS paper would be a bit pointless, given that they are already in the AAS submission and it would never fit into 1k words.

:+1: yes that's right. We also don't allow scientific results to be published in JOSS :-)

I hope that is okay btw, as I couldn't find any guidelines on this.

I think you've got this right. @crawfordsm and I are in the process of drafting some guidelines that should assist future authors.

fonnesbeck commented 5 years ago

@1313e the test suite fails on both Linux and macOS for me. pytest does not like the arguments:

pytest: error: unrecognized arguments: --mpl --pep8

fonnesbeck commented 5 years ago

I'm having problems running the example model. See 1313e/PRISM#7

1313e commented 5 years ago

@1313e the test suite fails on both Linux and macOS for me. pytest does not like the arguments:
pytest: error: unrecognized arguments: --mpl --pep8

@fonnesbeck You will have to install all requirements in requirements_dev.txt before running the pytests. I could put that in the README for clarity, if required.

fonnesbeck commented 5 years ago

Yes, at the moment the install instructions do not mention requirements_dev. Thanks.

1313e commented 5 years ago

Yes, at the moment the install instructions do not mention requirements_dev. Thanks.

@fonnesbeck Alright, I have added it to the README. I have also "fixed" 1313e/PRISM#7.

fonnesbeck commented 5 years ago

Tests now run after installing the dev requirements, but the tests fail.

fonnesbeck commented 5 years ago

The submission satisfies all of the evaluation criteria. Once the issue above is addressed, I'm happy to recommend ACCEPT.

1313e commented 5 years ago

@fonnesbeck I have solved the problem with 1313e/PRISM#8, and also improved the README and docs on the topic of running the pytests.

@arokem Do you have an idea how I can fix the formatting error that whedon gives when compiling the paper.md? It formats my last name incorrectly in the footer of every page (I already noticed that immediately after submission, but had not asked about it yet).

arfon commented 5 years ago

@1313e - I’ll have to fix the name issue manually at the end (the automated proofs will continue to have this problem)

fonnesbeck commented 5 years ago

I am now able to run the test suite without issue. There are no other outstanding problems, so I recommend ACCEPT. cc @arfon

arfon commented 5 years ago

@arokem - let me know when you're ready to accept this - I'll have to manually process this one to deal with Whedon's poor handling of van der Velden as a last name.

arokem commented 5 years ago

@whedon generate pdf

whedon commented 5 years ago

Attempting PDF compilation. Reticulating splines etc...

whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

arokem commented 5 years ago

@1313e

Two comments on the manuscript:

The first is that the link to the software is not necessary. As you can see in the rendered PDF, there will be a link to your GitHub repo on the first page of the manuscript. This is not a required fix.
Could you please add the DOI for the Raftery et al. paper? (DOI: 10.1080/01621459.1995.10476529). This one is required.

1313e commented 5 years ago

Could you please add the DOI for the Raftery et al. paper? (DOI: 10.1080/01621459.1995.10476529). This one is required.

Alright, fixed that. I had actually never been able to find a DOI for that paper, so thanks for finding that. :)

@arokem - let me know when you're ready to accept this - I'll have to manually process this one to deal with Whedon's poor handling of van der Velden as a last name.

I guess we will have to wait with accepting the paper until the AAS Journals submission has been reviewed, right?

arfon commented 5 years ago

I guess we will have to wait with accepting the paper until the AAS Journals submission has been reviewed, right?

Yes, I think that's right. @crawfordsm - does that sound about right to you?

arokem commented 5 years ago

@whedon generate pdf

whedon commented 5 years ago

Attempting PDF compilation. Reticulating splines etc...

whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

arokem commented 5 years ago

OK - I believe that this manuscript is ready to be accepted from our end.

@arfon: should @1313e create a Zenodo archive with the code as it is now, or wait until the AAS journal review is also resolved?

arfon commented 5 years ago

@arfon: should @1313e create a Zenodo archive with the code as it is now, or wait until the AAS journal review is also resolved?

I think it's OK to create the archive now.

1313e commented 5 years ago

Alright, should I make an archive of the current state of the master branch?

arfon commented 5 years ago

Yes please!

1313e commented 5 years ago

Alright, I had to get back out of my bed for it, but here is the zenodo DOI: 10.5281/zenodo.2572736

arfon commented 5 years ago

@whedon set 10.5281/zenodo.2572736 as archive

whedon commented 5 years ago

OK. 10.5281/zenodo.2572736 is the archive.

1313e commented 5 years ago

So, I guess all that is remaining is linking the JOSS submission with the AAS Journals submission (which is necessary anyway for the last paragraph) and manually adjust the PDF to have my last name spelled correctly?

Btw, I am actually wondering why whedon is writing my last name this weird. I have seen my name being spelled incorrectly in many different ways before, as most other countries do not understand that a last name can contain multiple words which should not be capitalized, but I have never seen it being spelled as "Van Velden" before.

Fun fact: I have seen no system thus far (besides those in my home country obviously) that can spell it correctly and still understand that my name should be alphabetically sorted on "Velden" rather than "van". For my name it does not matter that much given how close the two words are together alphabetically speaking, but there are many cases in which this is not true.

arokem commented 5 years ago

I must be missing something. Isn't your last name already spelled correctly here?

1313e commented 5 years ago

@arokem Take a look at the footer of every page.

arfon commented 5 years ago

Just to (re)confirm. When we finally accept this paper, I will fix this manually. The library we use for handling firstnames/lastnames can't handle van der so I have to fix papers by hand when this comes up.

arokem commented 5 years ago

Gotcha. Sorry I missed that!

1313e commented 5 years ago

@crawfordsm @arfon @arokem Alright, so I am currently in the process of revising the AAS Journals submission. One of the things that still needs to be done, is linking the JOSS submission and the AAS Journals submission together. However, as neither submission will get accepted before the other, no DOI exists for either submission. This makes it hard for me to cite one submission in the other, which is necessary for both.

So, therefore, I wanted to ask: Any ideas on how I should cite either submission in the other?

PS: I coincidentally went on a three week holiday the day that I received the referee report from AAS Journals, which is why it took until now to get to it.

arfon commented 5 years ago

@whedon generate pdf

openjournals / joss-reviews