[REVIEW]: Octopus-sensing: A Python library for human behavior studies

whedon commented 2 years ago

Submitting author: !--author-handle-->@nastaran62@ajstewartlang<!--end-editor-- Reviewers: @ixjlyons, @peircej Archive: 10.5281/zenodo.6350710

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/771f1e646589e0981d21be4e002fd28f"><img src="https://joss.theoj.org/papers/771f1e646589e0981d21be4e002fd28f/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/771f1e646589e0981d21be4e002fd28f/status.svg)](https://joss.theoj.org/papers/771f1e646589e0981d21be4e002fd28f)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ixjlyons & @peircej, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @ajstewartlang know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @ixjlyons

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@nastaran62) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @peircej

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@nastaran62) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

whedon commented 2 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @ixjlyons, @peircej it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

whedon commented 2 years ago

Wordcount for paper.md is 821

whedon commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.biopsycho.2010.03.010 is OK
- 10.1037/pspi0000337 is OK
- 10.3390/brainsci10020085 is OK
- 10.1016/j.imu.2020.100372 is OK
- 10.5334/pb-46-1-2-99 is OK
- 10.1016/j.ijhcs.2018.10.003 is OK
- 10.1186/s40561-018-0080-z is OK
- 10.1109/ACCESS.2020.3007109 is OK
- 10.1109/TCYB.2020.2974688 is OK
- 10.3390/s18072074 is OK
- 10.1109/T-AFFC.2011.15 is OK
- 10.1016/j.entcs.2019.04.009 is OK
- 10.1007/s10111-020-00641-0 is OK
- 10.1111/jcal.12590 is OK
- 10.3390/s20030592 is OK
- 10.1109/COMST.2017.2731979 is OK

MISSING DOIs

- None

INVALID DOIs

- None

whedon commented 2 years ago

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.12 s (717.0 files/s, 78467.9 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Python                           57           1078           2222           2979
XML                               3              0              0           1538
reStructuredText                 16            313            191            661
TeX                               1             20              0            171
Markdown                          2             29              0             90
Bourne Shell                      1             10              5             68
YAML                              2             17              5             68
TOML                              1              5              5             40
MATLAB                            1              3              0             31
DOS Batch                         1              8              1             26
make                              1              7             13             20
Bourne Again Shell                1              0              0              5
INI                               1              0              0              2
--------------------------------------------------------------------------------
SUM:                             88           1490           2442           5699
--------------------------------------------------------------------------------

Statistical information for the repository '77b353a1d810a5fb0b10d45d' was
gathered on 2022/01/10.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Aidin Gharibnavaz               28          1901            306           12.12
Nastaran                        76          8435           5145           74.56
Nastaran Saffaryazdi             7          1951            474           13.31
nastaran62                       1             1              1            0.01

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Aidin Gharibnavaz          1412           74.3          7.4               20.54
Nastaran                   4840           57.4          4.7               10.89
Nastaran Saffaryazdi         41            2.1         22.5               12.20

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 2 years ago

:wave: @peircej, please update us on how your review is going (this is an automated reminder).

whedon commented 2 years ago

:wave: @ixjlyons, please update us on how your review is going (this is an automated reminder).

peircej commented 2 years ago

I'm able to tick "yes" to most of the requirements already. 🥳 A couple of things remain

Community guidelines: at the moment I can't see any information about how users/contributors should get involved with the project. There is the GitHub issues list, which has a single closed issue. There's a page in the docs for developement but doesn't give any information about how to contribute - just information about how to pip install. I'd recommend the authors provide a line of information in the readme.md and also CONTRIBUTING file with guidelines

State of the Field: I can't see anything in the manuscript that considers other related packages. There are many packages around BCI using Python. There's Lab Streaming Layer for combining and time-stamping data from devices. There's PsychoPy for stimulus delivery. Maybe the authors should mention these and their relationships and/or their reasons for not integrating/extending those existing packages.

ixjlyons commented 2 years ago

Overall, Octopus Sensing appears to be a nice library for getting up and running with experiments requiring simultaneous recording from multiple sensors. I came across a few (mostly minor) issues described below, but otherwise I recommend accepting.

Minor issues

These are a few issues I noticed that might be good to have fixed prior to publication.

Documentation

Typo: README, link text "Otopus Sensing Website"
Broken link: Quick Start doc page link to "Development Guide"
sudo for running init.sh doesn't seem necessary (Quick Start doc) and should be avoided if possible.
Typo: docs tutorial section 3, first paragraph: "overal" -> "overall", "Uniti" -> "Unity", "taht" -> "that"
Need another troubleshooting step for PyGObject? May want to refer to their docs. I had to pre-install a couple system packages to get pygobject to install successfully.
Docs tutorial, section 9 header is ill-formatted.
Broken link: Octopus Sensing Visualizer page, note: reference to preprocessing module is broken
Stimuli API doc page is empty.

Code

init_script.sh points to nastaran62/octopus-sensing

Paper, JOSS checklist

Only first two paper authors are in the commit history. Maybe you could describe author contributions here in the review issue - I'll leave that to editor discretion.
Last sentence of second paragraph: no space after "Octopus Sensing".
Last sentence of third paragraph: could use grammar rework - unclear what "They" refers to (researchers or sensors?), particularly with phrases "making intelligent devices" and "making interactive robots".
I think the paper and documentation might benefit from a brief description of the methods used to provide simultaneous, synchronous recording from multiple sensors (i.e. process-based device interface). This could help convey the level of timing precision and accuracy to be expected.
Community guidelines appear to be missing.

Other Comments

These are some general comments on the software, documentation, etc. They're intended to be suggestions for further improvements but I don't consider them necessary for publication.

Lots of dependencies pulled in for devices I may or may not have. You might consider using something like extras_require for extra devices (not sure how that would work with Poetry).
In early parts of the tutorial, you might want to include imports to demonstrate how the package is organized.
Overall I think it could be helpful for the examples to run out-of-the-box if at all possible, whether it's by using only "standard" devices like camera streaming and audio that are expected to work without special hardware, or simulated devices just for demonstration. Being able to run an example without modification is an important step in evaluating software. As it is now, I don't think I can run any of the examples without at least some modifications. The endpoint_example only required setting up a directory with images - perhaps a small example image stack could be included in the repository.
In the tutorials, diagrams showing how things are connected would be helpful to understand the various network entities involved.
Docs tutorial, section 5: a link when referring to API for questionnaire and windows would be good. In general, links to API documentation are very helpful in tutorials/examples.

nastaran62 commented 2 years ago

@whedon commands

whedon commented 2 years ago

Here are some things you can ask me to do:

# List Whedon's capabilities
@whedon commands

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Ask Whedon to check the references for missing DOIs
@whedon check references

# Ask Whedon to check repository statistics for the submitted software
@whedon check repository

nastaran62 commented 2 years ago

@whedon check references

whedon commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.biopsycho.2010.03.010 is OK
- 10.1037/pspi0000337 is OK
- 10.3390/brainsci10020085 is OK
- 10.1016/j.imu.2020.100372 is OK
- 10.5334/pb-46-1-2-99 is OK
- 10.1016/j.ijhcs.2018.10.003 is OK
- 10.1186/s40561-018-0080-z is OK
- 10.1109/ACCESS.2020.3007109 is OK
- 10.1109/TCYB.2020.2974688 is OK
- 10.3390/s18072074 is OK
- 10.1109/T-AFFC.2011.15 is OK
- 10.1016/j.entcs.2019.04.009 is OK
- 10.1007/s10111-020-00641-0 is OK
- 10.1111/jcal.12590 is OK
- 10.3390/s20030592 is OK
- 10.1109/COMST.2017.2731979 is OK

MISSING DOIs

- 10.1109/taffc.2019.2902379 may be a valid DOI for title: Adapting software with affective computing: a systematic review

INVALID DOIs

- 10.1145/3490099.3511148 is INVALID

whedon commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.biopsycho.2010.03.010 is OK
- 10.1037/pspi0000337 is OK
- 10.3390/brainsci10020085 is OK
- 10.1016/j.imu.2020.100372 is OK
- 10.5334/pb-46-1-2-99 is OK
- 10.1016/j.ijhcs.2018.10.003 is OK
- 10.1186/s40561-018-0080-z is OK
- 10.1109/ACCESS.2020.3007109 is OK
- 10.1109/TCYB.2020.2974688 is OK
- 10.3390/s18072074 is OK
- 10.1109/T-AFFC.2011.15 is OK
- 10.1016/j.entcs.2019.04.009 is OK
- 10.1007/s10111-020-00641-0 is OK
- 10.1111/jcal.12590 is OK
- 10.3390/s20030592 is OK
- 10.1109/COMST.2017.2731979 is OK

MISSING DOIs

- 10.1109/taffc.2019.2902379 may be a valid DOI for title: Adapting software with affective computing: a systematic review

INVALID DOIs

- 10.1145/3490099.3511148 is INVALID

nastaran62 commented 2 years ago

@whedon check references

whedon commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.biopsycho.2010.03.010 is OK
- 10.1037/pspi0000337 is OK
- 10.3390/brainsci10020085 is OK
- 10.1016/j.imu.2020.100372 is OK
- 10.5334/pb-46-1-2-99 is OK
- 10.1016/j.ijhcs.2018.10.003 is OK
- 10.1186/s40561-018-0080-z is OK
- 10.1109/ACCESS.2020.3007109 is OK
- 10.1109/TCYB.2020.2974688 is OK
- 10.3390/s18072074 is OK
- 10.1109/T-AFFC.2011.15 is OK
- 10.1016/j.entcs.2019.04.009 is OK
- 10.1007/s10111-020-00641-0 is OK
- 10.1111/jcal.12590 is OK
- 10.3390/s20030592 is OK
- 10.1109/COMST.2017.2731979 is OK
- 10.1109/TAFFC.2019.2902379 is OK

MISSING DOIs

- None

INVALID DOIs

- None

nastaran62 commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

nastaran62 commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

nastaran62 commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

nastaran62 commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

nastaran62 commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

PDF failed to compile for issue #4045 with the following error:

 Error reading bibliography file paper.bib:
(line 126, column 3):
unexpected "d"
expecting space, ",", white space or "}"
Looks like we failed to compile the PDF

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

PDF failed to compile for issue #4045 with the following error:

 Error reading bibliography file paper.bib:
(line 128, column 3):
unexpected "p"
expecting space, ",", white space or "}"
Looks like we failed to compile the PDF

nastaran62 commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

nastaran62 commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

nastaran62 commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

nastaran62 commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

openjournals / joss-reviews