search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
722 stars 38 forks source link

[REVIEW]: DrWatson: the perfect sidekick for your scientific inquiries #2673

Closed whedon closed 4 years ago

whedon commented 4 years ago

Submitting author: @tamasgal (Tamás Gál) Repository: https://github.com/JuliaDynamics/DrWatson.jl Version: 1.16.3 Editor: @dpsanders Reviewers: @jpfairbanks, @kescobo, @apdavison Archive: 10.5281/zenodo.4133570

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/58375662785c6e80e2849fc2165471aa"><img src="https://joss.theoj.org/papers/58375662785c6e80e2849fc2165471aa/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/58375662785c6e80e2849fc2165471aa/status.svg)](https://joss.theoj.org/papers/58375662785c6e80e2849fc2165471aa)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@jpfairbanks & @kescobo, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @dpsanders know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @jpfairbanks

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @kescobo

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @apdavison

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 4 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @jpfairbanks, @kescobo it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
dpsanders commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

PDF failed to compile for issue #2673 with the following error:

sh: 0: getcwd() failed: No such file or directory pandoc: 10.21105.joss.02673.pdf: openBinaryFile: does not exist (No such file or directory) Looks like we failed to compile the PDF

whedon commented 4 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.jsc.2003.04.002 is OK
- doi:10.1201/9781315373461-3 is OK
- doi:10.5063/F1GF0RF6 is OK
- 10.6084/m9.figshare.3115156.v2 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 4 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

dpsanders commented 4 years ago

👋 Hi @jpfairbanks and @kescobo. Thank you for agreeing to review this submission. Please work your way through the checklist above of items. Please feel free to raise technical issues on the repository, and procedural questions here. Thanks!

kescobo commented 4 years ago

Does JOSS have the concept of "corresponding author" and/or author order conventions?

dpsanders commented 4 years ago

I would say no, that's up to the authors themselves.

kescobo commented 4 years ago

I've completed my review - pending the changes requested here, I'm happy to sign off.

jpfairbanks commented 4 years ago

So I have a question about scope. Is the JOSS paper format intended to be a complete description of the code? How much of the "explaining how to use the software" should be delegated to the docs (which will stay up to date).

dpsanders commented 4 years ago

Thanks @kescobo!

@jpfairbanks: That's a great question. The JOSS paper is definitely not a complete description of the code.

There should be one or a few code snippets giving a feeling for how to use the code, to explain the goals etc. of the software.

The docs should have a complete API description etc.

dpsanders commented 4 years ago

@jpfairbanks You can find review criteria here:

https://joss.readthedocs.io/en/latest/review_criteria.html

dpsanders commented 4 years ago

The idea is for the paper to be relatively short, with the main documentation in the software repository itself.

jpfairbanks commented 4 years ago

Thanks, I had read some of that language above in this thread and got confused because it talks about API docs, but I’m supposed to check that the API is documented in the actual manual. Not that the paper contains docs. This now makes sense. BTW I can’t mark the checkboxes in the checklist. Am I supposed to have different repo privs or something?

dpsanders commented 4 years ago

OK, let me know if the wording can be improved.

I believe that you should have got an invitation to the JOSS account giving you those privileges?

Datseris commented 4 years ago

Does JOSS have the concept of "corresponding author" and/or author order conventions?

For this paper you can consider myself the corresponding author (if need be).

jpfairbanks commented 4 years ago

I think that this is a great tool and a great write-up that sufficiently introduces the software, situates it in the relevant prior art and motivates me to adopt the software based on claimed benefits. The benefits of better simulation data management is obvious to anyone who has analyzed the performance, accuracy, or predictions of a computer simulation. There is no empirical evidence of efficiency or utility presented, but I am not sure that publication of this material requires that kind of empirical validation and defer to the editor's judgement.

The biggest issue from a methodological perspective is that Dr. Watson avoids the use of databases for managing the data collected during the experiments. The current practice for simulation data management is to create files with paths that encode information about the parameters and with contents that encode information about the results. Dr. Watson automates and formalizes these ad-hoc techniques. In that automating and formalizing ad hoc techniques can improve the reliability of the methods, this is good. But as a field, computer science has solved this data management problem with an entirely different technique.

Relational Database Management Systems (RDBMS) are the standard technology designed for managing data when you need persistence and correctness with the ability to store data quickly and query against it later. RDBMSes support flexible query languages for answering complex, ad-hoc queries about the data and indexing to make it fast. Why does Dr. Watson not use a database for storing this data? The paper should address this approach and why it was not selected. The current approach for simulation data management is ad hoc file naming conventions and folder structures containing structured or semi-structured data in the output files. So any solution that starts by improving this solution is welcome. However for more fruitful analysis of the resulting data, a proper data ingestion with ETL into an SQL database would allow for more rich querying. This could lead to easier feature development down the road when it comes to experiment analysis.

This concern could be addressed by identifying previous attempts to use RDBMSes in simulation data management and explaining why they failed, or discussing this path as a viable approach that has not yet been explored, and leaving integration with existing RDBMS techniques for future work, once the RDBMS ecosystem in Julia has matured.

Datseris commented 4 years ago

Thank you for the review @jpfairbanks , CaosDB is a RDBMS (I believe, although the creators name it Scientific Data Management System), and has been developed by scientists for scientists. We plan to integrate DrWatson with CaosDB.jl once a sufficiently stable Julia implementation exists. I believe once such an integration is in order, your concerns about the lack of data managing capabilities of DrWatson will be addressed. However, I do think this will take substantial amount of effort and thus it is unlikely that we are able to do it during the period of this review.

jpfairbanks commented 4 years ago

That sounds great, then I would request that the article describe the limitation of DrWatson for organizing or querying the resulting data and that you would use DrWatson for tracking the data generation processes and then ETL it into CaosDB for querying. Describing that as future work would be appropriate.

Datseris commented 4 years ago

Fair point, addressed in https://github.com/JuliaDynamics/DrWatson.jl/commit/bbe9a83dffc0e13d96285c8713db93ac1b695e89 .

Datseris commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Datseris commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Datseris commented 4 years ago

@dpsanders we thank both reviewers for the constructive and swift review. In the latest version of the paper (generated above) we address all raised concerns.

kescobo commented 4 years ago

@jpfairbanks did you manage to find the invite to https://github.com/openjournals/joss-reviews/ ? Mine showed up in my notifications, but I also got an email: image

you should also be able to click this link: https://github.com/openjournals/joss-reviews/invitations

jpfairbanks commented 4 years ago

I always forget that github invitations require interaction with email.

jpfairbanks commented 4 years ago

Please note that DrWatson is not a data management system and provides only basic data management functionality that remains self-contained in a single scientific project. One of our main future goals is to integrate DrWatson with a Relational Data Management System, specifically CaosDB (Fitschen et al., 2019), which had been developed specifically to handle large data bases connecting several scientific projects. 👍

dpsanders commented 4 years ago

@whedon add @apdavison as reviewer

whedon commented 4 years ago

OK, @apdavison is now a reviewer

dpsanders commented 4 years ago

Thanks @apdavison for agreeing to review this as well. Dr. Andrew Davison is one of the main authors of the Python sumatra package.

Datseris commented 4 years ago

Hello, may we please ask for an update here? @apdavison

apdavison commented 4 years ago

My apologies for the delay, I'll try to finish the review this week.

apdavison commented 4 years ago

@dpsanders I can't edit the checklist. When I click to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations it says the invitation has expired.

dpsanders commented 4 years ago

@whedon commands

whedon commented 4 years ago

Here are some things you can ask me to do:

# List all of Whedon's capabilities
@whedon commands

# Assign a GitHub user as the sole reviewer of this submission
@whedon assign @username as reviewer

# Add a GitHub user to the reviewers of this submission
@whedon add @username as reviewer

# Re-invite a reviewer (if they can't update checklists)
@whedon re-invite @username as reviewer

# Remove a GitHub user from the reviewers of this submission
@whedon remove @username as reviewer

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

# Change editorial assignment
@whedon assign @username as editor

# Set the software archive DOI at the top of the issue e.g.
@whedon set 10.0000/zenodo.00000 as archive

# Set the software version at the top of the issue e.g.
@whedon set v1.0.1 as version

# Open the review issue
@whedon start review

EDITORIAL TASKS

# All commands can be run on a non-default branch, to do this pass a custom 
# branch name by following the command with `from branch custom-branch-name`.
# For example:

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Remind an author or reviewer to return to a review after a
# certain period of time (supported units days and weeks)
@whedon remind @reviewer in 2 weeks

# Ask Whedon to do a dry run of accepting the paper and depositing with Crossref
@whedon accept

# Ask Whedon to check the references for missing DOIs
@whedon check references

# Ask Whedon to check repository statistics for the submitted software
@whedon check repository

EiC TASKS

# Invite an editor to edit a submission (sending them an email)
@whedon invite @editor as editor

# Reject a paper
@whedon reject

# Withdraw a paper
@whedon withdraw

# Ask Whedon to actually accept the paper and deposit with Crossref
@whedon accept deposit=true
dpsanders commented 4 years ago

@whedon re-invite @apdavison as reviewer

whedon commented 4 years ago

OK, the reviewer has been re-invited.

@apdavison please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

dpsanders commented 4 years ago

@apdavison I have re-invited you. Please let met know if it still doesn't work. Thanks!

apdavison commented 4 years ago

This seems like a nice toolkit for Julia users, which brings together a lot of functionality that would otherwise have to be assembled from several sources. I have the impression that a lot of the functionality relies on the modern design of Julia (in particular the package manager), and would be more difficult to implement in "messier" language ecosystems like Python's. The paper is well structured and clearly written.

I don't have any previous experience with Julia, so have not tried installing or using the package, but I think we can take the word of @kescobo and @jpfairbanks that this works!

My list of comments is not extensive, so I include it here rather than creating a new issue in the DrWatson repository.

Datseris commented 4 years ago

Thank you @apdavison for the quick response and your review. We have addressed your comments by:

  1. Adding a "Support & Contributing" section in the docs.
  2. Improving the introduction of sumatra
  3. Mentioning ReproZip (but not ActivePapers because contrary to its name it doesn't feel active enough of a repository to us)
  4. Addressing typos
Datseris commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

dpsanders commented 4 years ago

@Datseris I agree with @apdavison that ActivePapers should be cited, even if it is not currently "active".

Datseris commented 4 years ago

Alright, we have added the citation to it in the latest version.

Datseris commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

dpsanders commented 4 years ago

@Datseris @tamasgal I sent a few minor corrections to the paper.

Once that's done I think the paper is ready to accept! Please archive it at Zenodo and let me know the version and DOI. You need to make sure that the title and author info in Zenodo match the paper exactly. Thanks!

Datseris commented 4 years ago

Hi @dpsanders thanks for the improvements to the paper.

Here is the Zenodo DOI: 10.5281/zenodo.4133570

You need to make sure that the title and author info in Zenodo match the paper exactly.

The only problem is, I don't know how to do this... I didn't find a way in the website to alter the archived Zenodo information (and Zenodo archives GitHub repositories automatically). Do I have to make a manual Zenodo entry?

kescobo commented 4 years ago

@Datseris once you've linked the repo to Zenodo, every time you make a release (a github release, which may or may not happen automatically with your julia release, depending on your bots) it will automatically archive it