[REVIEW]: experDesign: helping performing experiments on batches

whedon commented 3 years ago

Submitting author: @llrs (Lluís Revilla Sancho) Repository: https://github.com/llrs/experDesign Version: v0.1.1 Editor: @lpantano Reviewer: @abartlett004, @stemangiola Archive: 10.5281/zenodo.5718051

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/57230bf58dae2e5efd80b72d155257c2"><img src="https://joss.theoj.org/papers/57230bf58dae2e5efd80b72d155257c2/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/57230bf58dae2e5efd80b72d155257c2/status.svg)](https://joss.theoj.org/papers/57230bf58dae2e5efd80b72d155257c2)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@abartlett004 & @stemangiola, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @lpantano know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @abartlett004

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@llrs) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @stemangiola

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[ ] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@llrs) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[ ] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[ ] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @abartlett004, @stemangiola it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

whedon commented 3 years ago

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.03 s (1655.0 files/s, 84174.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
R                               36            146            420            813
Markdown                         6            108              0            368
YAML                             5             39             10            175
JSON                             1              0              0            154
Rmd                              2             83            173             54
-------------------------------------------------------------------------------
SUM:                            50            376            603           1564
-------------------------------------------------------------------------------

Statistical information for the repository 'b2f7cfb1c184ae807aae4eda' was
gathered on 2021/06/11.
No commited files with the specified extensions were found.

whedon commented 3 years ago

PDF failed to compile for issue #3358 with the following error:

 Can't find any papers to compile :-(

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago

:wave: @abartlett004, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @stemangiola, please update us on how your review is going (this is an automated reminder).

stemangiola commented 3 years ago

Hello @llrs,

I have read the article and done my review. I attach it as a WORD files with comments.

Please also have a look at my wishlist above, for the requirements that have not been ticked.

10.21105.joss.03358.docx

llrs commented 3 years ago

Many thanks @stemangiola for all the comments and suggestions. I have opened an issue on the repository to track the points raised. I'll wait until the second reviewer submit the review to address them.

abartlett004 commented 3 years ago

I have also completed my review. I have added my comments to @stemangiola 's Word document and attached it here. I will also add comments regarding my unmarked items to the issue in the repository.

10.21105.joss.03358_ALB edits.docx

llrs commented 3 years ago

Many thanks @abartlett004 for your detailed comments and your review! I will start editing the manuscript to reduce duplicated content, add the missing section statement of need, make it more precise and at the same time add more content: code examples, function definition and comparison with other methods.

lpantano commented 3 years ago

@llrs, any updates on this? thanks!

llrs commented 3 years ago

Yes, sorry for the long silence. I have an updated version of the manuscript almost ready to be updated again. There weren't many (any) issues with the software.

However, I have a doubt about the "Substantial scholarly effort" criteria. It was approved for review with a comment about the small size of the software. Both reviewers didn't mark the "Substantial scholarly effort" (I think because the lines of code are not above the threshold) but I haven't found any clue on which direction the package would need to be expanded or if the manuscript would be approved to publication without more substantial effort on the software. Many thanks for checking the status.

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 3 years ago

@lpantano Submitting this new version for further comments and if decided to publication.

Here are some comments on the review @stemangiola.:

The license is both on the DESCRIPTION of the package and on the LICENSE.md and LICENSE file on the root directory with an approved OSI license in plain text.
I have provided examples on the paper about how to use the program. In addition to reviewing the code examples.
The functions are now also described on the manuscript beside the package own documentation. Hope it is now clear how do they work now.

Both of you @stemangiola and @abartlett004 raised some concerns regarding the scholarly effort, statement of need and quality of writing. See below for my comments:

Substantial scholarly effort: The amount of work on the software was of 3 months between thinking about the implementation and implementing in few lines of code. If there is anything lacking to the software I would gladly add it to extend and make it more useful. There are some ideas on the issue tracker but I don't think they are important, and they might blur the focus of the package.
Statement of need: A new section has been added with a statement of need.
Quality of writing: The manuscript has been through-fully checked and reorganized for better readability. Apologies for all the mistakes it had.

lpantano commented 3 years ago

Thank you, @llrs for your comments. I will wait for the reviewers to come back with their feedback.

I can comment on the "Substantial scholarly effort". It is true we have a cutoff of 1000 lines, but since this is more than 500, I would recommend the reviewers to evaluate whether this tools would be useful and cited in the future, and recommend any addition that would make a difference in that sense.

Thanks everybody.

lpantano commented 3 years ago

@stemangiola , @abartlett004, could you give us a timeframe for reviewing the final items that are unchecked in the list above. If something is still missing, you can post the comments here and the authors can address them.

lpantano commented 3 years ago

@llrs, the one thing that I couldn't find is some kind of score telling you how good or bad is the final design. I can imagine that there are cases where is not ideal, like in one of your example, and it would be good to have some score or warning telling you that even trying to randomize the design, the design will be biased by some degree. Let me know your thoughts. Maybe it is already there and it only needs a better place in the documentation. thanks!

llrs commented 3 years ago

@lpantano such score is already available on the package. One can explore how good a fit is with check_index but this is only between subsets, not among all the combinations. The closer to 0, the better a subset is. I will update the documentation of the function with more comments pointing users to increase the iterations and mention the check_index to the vignette too. Thanks for the suggestion.

abartlett004 commented 3 years ago

Thanks for making the revisions! The paper overall looks much better to me now. The writing quality is much improved and the discussion and examples of other existing packages really helped me to understand why your package is substantial. My final comment is that I think you forgot to insert the flowchart regarding which functions to use. Other than that, everything looks good!

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 3 years ago

@abartlett004 Apologies, forgot to commit the flowchart (see the new updated version). Many thanks for your encouraging words and review, it has helped to show the value of the package.

lpantano commented 2 years ago

@stemangiola, could you give us an update on this? thanks!

lpantano commented 2 years ago

@stemangiola, are you still available to finish this?, thanks.

lpantano commented 2 years ago

Hi @llrs, sorry about the delay. I will finish the review from @stemangiola and try to close this in the next days. Thank you for your patience.

lpantano commented 2 years ago

@whedon check references from branch paper

whedon commented 2 years ago

Attempting to check references... from custom branch paper

whedon commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1038/nrg2825 is OK
- 10.15252/embj.201592958 is OK
- 10.1371/journal.pone.0017238 is OK
- 10.1186/1471-2164-13-689 is OK
- 10.1093/bioinformatics/btab159 is OK
- 10.1038/nmeth.3091 is OK
- 10.1037/met0000301 is OK

MISSING DOIs

- None

INVALID DOIs

- None

lpantano commented 2 years ago

Hi @llrs, I think the last modification would be related to the length of the paper. The idea is that the papers is 2-3 page long. You could point to all the examples in the paper to the documentation. If there are examples that are not in the docs, you can add it there. Maybe you can do a panel of 2-3 figures of the most important examples and leave the code out (mentioning where you can find that in the docs). Once this is re-formatted, we are good to accept the paper finally. Thank you!

llrs commented 2 years ago

Hi @lpantano, Thanks for editing the manuscript. I will just provide some links to where to find examples of said functionality and remove the code from the others tools, showing only a couple of figures comparing the solutions of each package.

llrs commented 2 years ago

@whedon generate pdf from branch paper

whedon commented 2 years ago

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 2 years ago

@whedon generate pdf from branch paper

whedon commented 2 years ago

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 2 years ago

@whedon generate pdf from branch paper

whedon commented 2 years ago

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 2 years ago

@whedon generate pdf from branch paper

whedon commented 2 years ago

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

openjournals / joss-reviews