whedon commented 4 years ago

Submitting author: @FlorianRhiem (Florian Rhiem) Repository: https://github.com/sciapp/sampledb Version: 0.17.0 Editor: @arfon Reviewer: @stuartcampbell, @dvanic Archive: 10.5281/zenodo.4529206

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/3769b983ef954e66bf7e3e7d064e7c7e"><img src="https://joss.theoj.org/papers/3769b983ef954e66bf7e3e7d064e7c7e/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/3769b983ef954e66bf7e3e7d064e7c7e/status.svg)](https://joss.theoj.org/papers/3769b983ef954e66bf7e3e7d064e7c7e)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@stuartcampbell & @dvanic, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @trallard know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @stuartcampbell

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@FlorianRhiem) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @dvanic

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@FlorianRhiem) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

whedon commented 4 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @eteq, @dvanic it looks like you're currently assigned to review this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

whedon commented 4 years ago

Reference check summary:

OK DOIs

- None

MISSING DOIs

- None

INVALID DOIs

- None

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

danielskatz commented 4 years ago

👋 @eteq, @dvanic - It doesn't look like there's been any progress on this review in about 3 1/2 weeks - Can everyone please check in and provide an estimate of when your reviews might be started/finished?

FlorianRhiem commented 4 years ago

@trallard SampleDB v0.9.0 has been released yesterday. I've updated the paper.json. Could you please run @whedon set v0.9.0 as version to update the version here?

trallard commented 4 years ago

@whedon set v0.9.0 as version

whedon commented 4 years ago

OK. v0.9.0 is the version.

trallard commented 4 years ago

@eteq and @dvanic any chance you could check in an give any updates re your review?

arfon commented 4 years ago

Dear authors and reviewers

We wanted to notify you that in light of the current COVID-19 pandemic, JOSS has decided to suspend submission of new manuscripts and to handle existing manuscripts (such as this one) on a "best efforts basis". We understand that you may need to attend to more pressing issues than completing a review or updating a repository in response to a review. If this is the case, a quick note indicating that you need to put a "pause" on your involvement with a review would be appreciated but is not required.

Thanks in advance for your understanding.

Arfon Smith, Editor in Chief, on behalf of the JOSS editorial team.

arfon commented 4 years ago

:wave: @eteq & @dvanic, just a friendly check-in to see how things are going with your reviews?

FlorianRhiem commented 4 years ago

@trallard I have just released SampleDB v0.10.0. Could you please run @whedon set v0.10.0 as version so that it is updated here? Thank you!

arfon commented 4 years ago

@whedon set v0.10.0 as version

whedon commented 4 years ago

OK. v0.10.0 is the version.

arfon commented 4 years ago

👋 @eteq & @dvanic - today we reopened JOSS for new submissions and are checking in on our existing reviews. Do you think you might be able to wrap up your reviews in the next 2-3 weeks?

trallard commented 4 years ago

Hi folks just checking in on the status as per Arfon's message. Any updates on the reviews @eteq @dvanic

dvanic commented 4 years ago

@whedon commands

whedon commented 4 years ago

Here are some things you can ask me to do:

# List Whedon's capabilities
@whedon commands

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Ask Whedon to check the references for missing DOIs
@whedon check references

# Ask Whedon to check repository statistics for the submitted software
@whedon check repository

dvanic commented 4 years ago

@whedon I need a new invitation:

whedon commented 4 years ago

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@whedon commands

dvanic commented 4 years ago

@whedon generate pdf

dvanic commented 4 years ago

@whedon check references

dvanic commented 4 years ago

@whedon check repository

whedon commented 4 years ago

Reference check summary:

OK DOIs

- None

MISSING DOIs

- None

INVALID DOIs

- None

whedon commented 4 years ago

Software report (experimental):

github.com/AlDanial/cloc v 1.84  T=1.34 s (345.2 files/s, 64768.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                         223           6529           3385          30001
JavaScript                      76           2128           2083          17229
CSS                             16             58             74          10764
HTML                            96            204             12           5965
SVG                              2              0              0           2959
JSON                            31              0              0           2604
reStructuredText                11            678           1092            391
YAML                             1              8             11            112
Markdown                         3             38              0            101
TeX                              1              0              0             28
Dockerfile                       1              9              7             20
-------------------------------------------------------------------------------
SUM:                           461           9652           6664          70174
-------------------------------------------------------------------------------

Statistical information for the repository '2107' was gathered on 2020/06/10.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Daniel Kaiser                    1            84             12            0.10
Dorothea Henkel                 56          5257           2195            7.54
Florian Rhiem                  360         74066          16361           91.49
peters@iff242                    5           720            141            0.87

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Daniel Kaiser                72           85.7         20.0                0.00
Dorothea Henkel            2187           41.6         37.2                9.37
Florian Rhiem             58985           79.6         27.9                6.16
peters@iff242               185           25.7         20.8                0.00

dvanic commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

dvanic commented 4 years ago

Review checklist for @dvanic

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@FlorianRhiem) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x ] Functionality: Have the functional claims of the software been confirmed?
- Note: not yet. There are challenges with documentation around setting up new users, groups and projects.
- Update: these have been addressed
- [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).

This is the area I struggled with the most: I could not identify clear documentation on how to set up a completely new instance of sampleDB at a new institution, add users, groups, projects, metadata, samples, instruments etc, and then how to proceed to a tracked lifecycle of samples and visualisations in Jupyter.

Please note I followed the "Getting Started" instructions in the GitHub repository, and while I could try to read the output options for all of the possible commands in docker exec sampledb env/bin/python -m sampledb help it's not clear what the optimal/recommended order of setup is. To enable wide adoption of this tool a case study of setting up a basic database is vital (because I genuinely think this is a very useful software, that really could be adopted beyond the institution in which it was developed!).

This has been addressed.

[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

API documentation is available. Due to not being able to set up or access a dummy database I was unable to test it.

[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

Yes, although the database could be of use beyond physics (for ex. in the life sciences, especially biology), and I'm not sure the provided description would enable life science users to see this.

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

Yes.

[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?

No. No mention of other packages is made.

[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[ x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

No scientific publications are cited - only references to software tools that underpin the database, and a metapaper around research data management.

Footnote

I'd like to personally apologize to @FlorianRhiem for the amount of time it's taken me to review this. While I'm fortunate enough to be in Australia, where COVID has not raged as widely as in many other parts of the world, our workload this semester has been horrific due to the move to online teaching/training (as well as the challenges of childcare and life). @FlorianRhiem I am SO sorry for how long this review has taken! And while I have made some suggestions, I think this is a very useful piece of software, and one we can potentially adapt to some of the work we do with researchers at the Sydney Informatics Hub/University of Sydney.

FlorianRhiem commented 4 years ago

Thank you for the review, Darya. I'm glad to hear that you think SampleDB could be useful for more researchers and I will try to fix these issues until the end of next week.

I think the most difficult points will be the documentation for more than the minimal setup and for the JupyterHub integration. For both there are dependencies to likely existing IT infrastructure (where should backups be stored, which volumes are mounted by the JupyterHub notebook servers, etc.), but I will try to keep the information as independent of these details as possible. Currently the online documentation is only updated for the tagged releases, so if that's alright with you, I will post drafts for these texts in the GitHub issues you opened, so that you can let me know whether something is missing or too vague.

danielskatz commented 4 years ago

@whedon re-invite @dvanic as reviewer

whedon commented 4 years ago

OK, the reviewer has been re-invited.

@dvanic please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

FlorianRhiem commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

FlorianRhiem commented 4 years ago

I have created an Administrator Guide with a new Getting Started guide, which walks readers through the first steps. I hope this solves the functionality and documentation issues for setting up SampleDB. If information on a particular step is still missing or too vague, please let me know!

I have added a Support section to the README and also recommend opening GitHub issues at the end of the Getting Started guide.

I have updated the paper to include a paragraph in which I discuss what I see as the largest difference between SampleDB and other applications with similar goals, in particular JuliaBase and BikaLIMS / Senaite, and which advantages and disadvantages result from this. In terms of goals, JuliaBase is the closest to SampleDB, but BikaLIMS / Senaite include sample management as a core feature as well and I think it's worth mentioning them as their approach of having pre-defined calculations might be something worth exploring in the future, for use cases where a data analysis in a Jupyter notebook might not be necessary.

I have also added a reference to The FAIR Guiding Principles for scientific data management and stewardship and updated the text to include more than just making the data accessible, as the motivation and reasoning behind the FAIR principles as presented in the paper match the motivation behind SampleDB as well.

I hope I have addressed the issues with SampleDB and the paper, either here or in the GitHub Issues of the project. Please let me know what I can improve and thank you again for taking the time to review the paper, particular given the current situation.

@trallard I've released SampleDB 0.11.0, could you please updated the version with whedon? Thank you!

trallard commented 4 years ago

@whedon set 0.11.0 as version

whedon commented 4 years ago

OK. 0.11.0 is the version.

trallard commented 4 years ago

@FlorianRhiem thanks for the ping I have updated the version but will have to rectify as the review moves forward.

-- @eteq, @dvanic how are your reviews coming along? Do you need any support from my side?

trallard commented 4 years ago

@eteq, @dvanic gentle ping again. Are there any blockers for your reviews, would it be possible to move them forward?

labarba commented 4 years ago

👋 hi everybody!

It looks like this review was progressing, but stalled for some reason. Can we have an update from @eteq, @dvanic on the status of their reviews? If you need more time, just let us know—times are tough!

FlorianRhiem commented 4 years ago

SampleDB 0.14 was released yesterday.

Is there any way I can help you with the review?

arfon commented 4 years ago

:wave: @trallard - it might be a good idea to try contacting our reviewers via email at this point? Many of our reviewers struggle with GitHub notifications and I'm wondering if this review has got lost in the noise :-/

dvanic commented 4 years ago

@arfon / @trallard - I have reviewed this and think the issued I identified have been resolved with the new documentation.

arfon commented 3 years ago

@trallard - I think we're going to need to find a second reviewer for this submission as it doesn't look like @eteq is going to be able to assist here.

arfon commented 3 years ago

:wave: @celliern @schnorr, would any of you be willing to provide a second review for this submission to JOSS?

The title of the submission is "SampleDB: A sample and measurement metadata database", which describes a Python package for managing experiments. Quoting from the SampleDB website:

SampleDB is a web-based sample and measurement metadata database developed at PGI and JCNS. Scientists can use SampleDB to store and retrieve information on samples, measurements and simulations, analyze them using Jupyter notebooks, track sample locations and responsibilities and view sample lifecycles. As a tool for reproducible and sustainable science, it facilitates long-term storage of research data with clear benefits to scientists, interfaces with dedicated JupyterHub instances and can be integrated in instrument systems to automate data entry. A schema system allows users to store and search even complex types of metadata, and the powerful permission system ensures that information stays secure while simplifying collaboration with external users.

Our request here would be that you review the software and short paper, following the JOSS checklist-driven review process.

Many thanks!

arfon commented 3 years ago

@arfon / @trallard - I have reviewed this and think the issued I identified have been resolved with the new documentation.

Thanks for your review here @dvanic. Could I ask you to check of any remaining items in your checklist at the top of the issue (assuming these criteria have now been met)?

arfon commented 3 years ago

👋 @DanNixon @stuartcampbell @cmbiwer, would any of you be willing to provide a second review for this submission to JOSS?

The title of the submission is "SampleDB: A sample and measurement metadata database", which describes a Python package for managing experiments. Quoting from the SampleDB website:

SampleDB is a web-based sample and measurement metadata database developed at PGI and JCNS. Scientists can use SampleDB to store and retrieve information on samples, measurements and simulations, analyze them using Jupyter notebooks, track sample locations and responsibilities and view sample lifecycles. As a tool for reproducible and sustainable science, it facilitates long-term storage of research data with clear benefits to scientists, interfaces with dedicated JupyterHub instances and can be integrated in instrument systems to automate data entry. A schema system allows users to store and search even complex types of metadata, and the powerful permission system ensures that information stays secure while simplifying collaboration with external users.

Our request here would be that you review the software and short paper, following the JOSS checklist-driven review process.

Many thanks!

stuartcampbell commented 3 years ago

Yes I'd be happy to help.

arfon commented 3 years ago

@whedon re-invite @stuartcampbell as reviewer

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@stuartcampbell please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

arfon commented 3 years ago

Yes I'd be happy to help.

Great! Many thanks! Please accept the invite in the post above to start. I've added a review checklist for you above too. Any questions, don't hesitate to ask.

stuartcampbell commented 3 years ago

@arfon I think I am done with my review. I have created a couple of issues over on the SampleDB Github site, but I don't think that they need to hold up this paper. They are more comments and nice to haves.

openjournals / joss-reviews

[REVIEW]: SampleDB: A sample and measurement metadata database #2107

Status

Reviewer instructions & questions

Review checklist for @stuartcampbell

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @dvanic

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @dvanic

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Footnote