search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
722 stars 38 forks source link

[REVIEW]: experDesign: helping performing experiments on batches #3358

Closed whedon closed 2 years ago

whedon commented 3 years ago

Submitting author: @llrs (Lluís Revilla Sancho) Repository: https://github.com/llrs/experDesign Version: v0.1.1 Editor: @lpantano Reviewer: @abartlett004, @stemangiola Archive: 10.5281/zenodo.5718051

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/57230bf58dae2e5efd80b72d155257c2"><img src="https://joss.theoj.org/papers/57230bf58dae2e5efd80b72d155257c2/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/57230bf58dae2e5efd80b72d155257c2/status.svg)](https://joss.theoj.org/papers/57230bf58dae2e5efd80b72d155257c2)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@abartlett004 & @stemangiola, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @lpantano know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @abartlett004

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @stemangiola

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @abartlett004, @stemangiola it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.03 s (1655.0 files/s, 84174.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
R                               36            146            420            813
Markdown                         6            108              0            368
YAML                             5             39             10            175
JSON                             1              0              0            154
Rmd                              2             83            173             54
-------------------------------------------------------------------------------
SUM:                            50            376            603           1564
-------------------------------------------------------------------------------

Statistical information for the repository 'b2f7cfb1c184ae807aae4eda' was
gathered on 2021/06/11.
No commited files with the specified extensions were found.
whedon commented 3 years ago

PDF failed to compile for issue #3358 with the following error: 

 Can't find any papers to compile :-(
llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago

:wave: @abartlett004, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @stemangiola, please update us on how your review is going (this is an automated reminder).

stemangiola commented 3 years ago

Hello @llrs,

I have read the article and done my review. I attach it as a WORD files with comments.

Please also have a look at my wishlist above, for the requirements that have not been ticked.

10.21105.joss.03358.docx

llrs commented 3 years ago

Many thanks @stemangiola for all the comments and suggestions. I have opened an issue on the repository to track the points raised. I'll wait until the second reviewer submit the review to address them.

abartlett004 commented 3 years ago

I have also completed my review. I have added my comments to @stemangiola 's Word document and attached it here. I will also add comments regarding my unmarked items to the issue in the repository.

10.21105.joss.03358_ALB edits.docx

llrs commented 3 years ago

Many thanks @abartlett004 for your detailed comments and your review! I will start editing the manuscript to reduce duplicated content, add the missing section statement of need, make it more precise and at the same time add more content: code examples, function definition and comparison with other methods.

lpantano commented 3 years ago

@llrs, any updates on this? thanks!

llrs commented 3 years ago

Yes, sorry for the long silence. I have an updated version of the manuscript almost ready to be updated again. There weren't many (any) issues with the software.

However, I have a doubt about the "Substantial scholarly effort" criteria. It was approved for review with a comment about the small size of the software. Both reviewers didn't mark the "Substantial scholarly effort" (I think because the lines of code are not above the threshold) but I haven't found any clue on which direction the package would need to be expanded or if the manuscript would be approved to publication without more substantial effort on the software. Many thanks for checking the status.

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 3 years ago

@lpantano Submitting this new version for further comments and if decided to publication.

Here are some comments on the review @stemangiola.:

Both of you @stemangiola and @abartlett004 raised some concerns regarding the scholarly effort, statement of need and quality of writing. See below for my comments:

lpantano commented 3 years ago

Thank you, @llrs for your comments. I will wait for the reviewers to come back with their feedback.

I can comment on the "Substantial scholarly effort". It is true we have a cutoff of 1000 lines, but since this is more than 500, I would recommend the reviewers to evaluate whether this tools would be useful and cited in the future, and recommend any addition that would make a difference in that sense.

Thanks everybody.

lpantano commented 3 years ago

@stemangiola , @abartlett004, could you give us a timeframe for reviewing the final items that are unchecked in the list above. If something is still missing, you can post the comments here and the authors can address them.

lpantano commented 3 years ago

@llrs, the one thing that I couldn't find is some kind of score telling you how good or bad is the final design. I can imagine that there are cases where is not ideal, like in one of your example, and it would be good to have some score or warning telling you that even trying to randomize the design, the design will be biased by some degree. Let me know your thoughts. Maybe it is already there and it only needs a better place in the documentation. thanks!

llrs commented 3 years ago

@lpantano such score is already available on the package. One can explore how good a fit is with check_index but this is only between subsets, not among all the combinations. The closer to 0, the better a subset is. I will update the documentation of the function with more comments pointing users to increase the iterations and mention the check_index to the vignette too. Thanks for the suggestion.

abartlett004 commented 3 years ago

Thanks for making the revisions! The paper overall looks much better to me now. The writing quality is much improved and the discussion and examples of other existing packages really helped me to understand why your package is substantial. My final comment is that I think you forgot to insert the flowchart regarding which functions to use. Other than that, everything looks good!

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 3 years ago

@abartlett004 Apologies, forgot to commit the flowchart (see the new updated version). Many thanks for your encouraging words and review, it has helped to show the value of the package.

lpantano commented 3 years ago

@stemangiola, could you give us an update on this? thanks!

lpantano commented 3 years ago

@stemangiola, are you still available to finish this?, thanks.

lpantano commented 3 years ago

Hi @llrs, sorry about the delay. I will finish the review from @stemangiola and try to close this in the next days. Thank you for your patience.

lpantano commented 3 years ago

@whedon check references from branch paper

whedon commented 3 years ago 
Attempting to check references... from custom branch paper
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1038/nrg2825 is OK
- 10.15252/embj.201592958 is OK
- 10.1371/journal.pone.0017238 is OK
- 10.1186/1471-2164-13-689 is OK
- 10.1093/bioinformatics/btab159 is OK
- 10.1038/nmeth.3091 is OK
- 10.1037/met0000301 is OK

MISSING DOIs

- None

INVALID DOIs

- None
lpantano commented 3 years ago

Hi @llrs, I think the last modification would be related to the length of the paper. The idea is that the papers is 2-3 page long. You could point to all the examples in the paper to the documentation. If there are examples that are not in the docs, you can add it there. Maybe you can do a panel of 2-3 figures of the most important examples and leave the code out (mentioning where you can find that in the docs). Once this is re-formatted, we are good to accept the paper finally. Thank you!

llrs commented 3 years ago

Hi @lpantano, Thanks for editing the manuscript. I will just provide some links to where to find examples of said functionality and remove the code from the others tools, showing only a couple of figures comparing the solutions of each package.

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

llrs commented 3 years ago

@whedon generate pdf from branch paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left: