openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
721 stars 38 forks source link

[REVIEW]: ReSurfEMG: A Python library for preprocessing and analysis of respiratory EMG. #5251

Closed editorialbot closed 1 year ago

editorialbot commented 1 year ago

Submitting author: @drcandacemakedamoore (Candace Makeda Moore) Repository: https://github.com/ReSurfEMG/ReSurfEMG Branch with paper.md (empty if default branch): Version: v0.1.3 Editor: !--editor-->@AJQuinn<!--end-editor-- Reviewers: @ixjlyons, @ajbaird Archive: 10.5281/zenodo.8429265

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/5f08d1f2bb717b7d05762296e37ded3d"><img src="https://joss.theoj.org/papers/5f08d1f2bb717b7d05762296e37ded3d/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/5f08d1f2bb717b7d05762296e37ded3d/status.svg)](https://joss.theoj.org/papers/5f08d1f2bb717b7d05762296e37ded3d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@marcoghislieri & @ixjlyons, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @AJQuinn know.

Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest

Checklists

📝 Checklist for @ixjlyons

📝 Checklist for @ajbaird

editorialbot commented 1 year ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 1 year ago
Software report:

github.com/AlDanial/cloc v 1.88  T=0.82 s (100.3 files/s, 258079.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Jupyter Notebook                38              0         195457           8679
Python                          12            690           1139           2616
CSS                              1            192             32            685
Markdown                         7            226              0            522
YAML                             6             36             23            233
TeX                              1             10              0            143
reStructuredText                12             84             66            128
Dockerfile                       1              2              0              4
JSON                             1              1              0              4
Bourne Shell                     1              0              0              3
DOS Batch                        1              0              0              2
SVG                              1              0              0              1
-------------------------------------------------------------------------------
SUM:                            82           1241         196717          13020
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository
editorialbot commented 1 year ago

Wordcount for paper.md is 871

editorialbot commented 1 year ago
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/S0921-884X(96)95190-5 is OK
- 10.1109/10.661154 is OK
- 10.1016/0013-4694(86)90163-X is OK
- 10.1046/j.1365-2842.1998.00242.x is OK
- 10.1007/s00421-010-1521-8 is OK
- 10.3758/s13428-020-01516-y is OK
- 10.21105/joss.04156 is OK

MISSING DOIs

- 10.1007/978-3-642-34546-3_71 may be a valid DOI for title: Towards Improving the Usability of Electromyographic Interfaces

INVALID DOIs

- None
editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

AJQuinn commented 1 year ago

👋🏼 @drcandacemakedamoore @marcoghislieri @ixjlyons this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#5251 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@ajquinn) if you have any questions/concerns.

AJQuinn commented 1 year ago

Hi @marcoghislieri @ixjlyons - its been around a week so I'd like to check in and make sure all is in place for you to get started on the review?

You can create a list of key points to assess for the review by commenting @editorialbot generate my checklist in this thread. Then the main review can take place over in ReSurfEMG's issues board - don't forget to tag this thread in any issues that you open so we can keep track.

Let me know if you have face any issues or have any questions.

Cheers

ixjlyons commented 1 year ago

Thanks @AJQuinn, no issues for me so far, just a delayed start.

~@editorialbot generate my checklist~

ixjlyons commented 1 year ago

Review checklist for @ixjlyons

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

AJQuinn commented 1 year ago

Hi @marcoghislieri - have you been able to get started on the checklist and the review? We're hoping to get the reviews in place over the next couple of weeks.

No problem if you'll need a bit more time, just let me know here or via and email.

AJQuinn commented 1 year ago

Hi @ixjlyons Thanks for filling out the review checklist.

Are there specific issues to be resolved before you can sign-off on the 'Installation' and 'Functionality' sections? Could you add a link to this thread in the relevant github issues so I can keep a track of them from here?

Cheers

ixjlyons commented 1 year ago

No particular issues so far, I just wasn't able to finish reviewing in one pass and I had to step away for a couple weeks. I should be able to revisit and finish up in the next few days.

AJQuinn commented 1 year ago

Hello @behinger, @ajbaird and @peterakirk!

This is a review thread for a Python package "ReSurfEMG: A Python library for preprocessing and analysis of respiratory EMG" at the Journal of Open Source Software. More details in the links in the top comment.

Review has started - but we need to find another reviewer to join us at short notice. Would any of you be willing to contribute a review for this submission for JOSS? We'd ideally like to find someone who could get started soon as the review has been going for a while now.

JOSS carries out checklist-driven reviews here in GitHub issues. Reviews follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

If you have any questions or queries before committing to a review please just let me know!

behinger commented 1 year ago

hi! unfortunately I am swamped right now - it will be better in 3-4 weeks only

ajbaird commented 1 year ago

I'm happy to help but what does the timeline look like. I can possibly finish a review in 2 weeks? If that timeline works let me know!

AJQuinn commented 1 year ago

Thank you both,

Around two weeks for the review would be great if that's possible @ajbaird - let me know and I'll set things up.

No problem @behinger, thanks for letting me know!

AJQuinn commented 1 year ago

@editorialbot remove @marcoghislieri from reviewers

editorialbot commented 1 year ago

@marcoghislieri removed from the reviewers list!

peterakirk commented 1 year ago

I'm swamped and just finished another JOSS review yesterday, so unfortunately I must decline. Best of luck!

AJQuinn commented 1 year ago

No problem, thanks for letting me know @peterakirk

@ajbaird - quick clarification! two weeks would be great, but fine if you need a bit longer as long as we have a broad idea in advance.

Thanks all

ajbaird commented 1 year ago

ok great, well lets say June 10th deadline and I should be able to reach that goal. Happy to help out!

AJQuinn commented 1 year ago

@editorialbot add @ajbaird as reviewer

editorialbot commented 1 year ago

@ajbaird added to the reviewers list!

AJQuinn commented 1 year ago

Super - thanks @ajbaird! let me know if you're going to be significantly later than the 10th.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread. You can find a full set of reviewer guidelines here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#5251 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

Please feel free to ping me if you have any questions/concerns.

ajbaird commented 1 year ago

Review checklist for @ajbaird

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

ixjlyons commented 1 year ago

Hi all, my sincere apologies for taking so long to get this review in. This one really got away from me. Anyway, here's my review.


Overall Comments

I think the purpose of the software is clear and described well in the paper. Having worked for some time in a related field, I can attest to lack of clarity in algorithm implementations and parameters. I have some overall feedback that I think could improve the repository, but otherwise it appears to be in good shape.

The main constructive feedback I have for ReSurfEMG is to spruce up the analysis notebooks a bit more, perhaps with some additional narrative explanations weaved around the code cells. I think most of them already do this pretty well, but some start to get heavy with code and sparser explanations toward the end. You might also consider changing the usage of widgets in the notebooks to update analysis/plots as user choices change (datapoint, number of samples, etc.) instead of having to go back to the widget and then run subsequent cells. My usual approach to this is to choose a default set of inputs, write out the analysis steps via functions with intermediate evaluations to walk through what's being done, then hook up the function(s) to widgets for interactivity. I consider these recommendations/suggestions, not mandatory for acceptance.

Paper

My only comments on the paper are minor:

Documentation

Installation notes:

Searching for notebook
Reading https://pypi.org/simple/notebook/
Downloading https://files.pythonhosted.org/packages/67/ae/fd5ce79fb604fb4a9bc582bfa24dd3b27d85cdcee5d646abb3f905deebfb/notebook-7.0.0b0-py3-none-any.whl#sha256=a73f295ad0e728d01af316ed04cadb39913da1e77647faf57a3c4f6b3844a303
Best match: notebook 7.0.0b0
Processing notebook-7.0.0b0-py3-none-any.whl
Installing notebook-7.0.0b0-py3-none-any.whl to /home/krlyons/tmp/ReSurfEMG/.venv/lib/python3.8/site-packages
error: [Errno 13] Permission denied: '/home/krlyons/tmp/ReSurfEMG/.venv/lib/python3.8/site-packages/notebook-7.0.0b0-py3.8.egg/notebook-7.0.0b0.data/data/share' -> '/home/krlyons/tmp/ReSurfEMG/.venv/lib/python3.8/site-packages/notebook-7.0.0b0-py3.8.egg/./share'

Due to the notes above, I'll leave the installation box unchecked for now, but I was ultimately able to get the package installed to run the analysis notebooks and try out ResurfEMG-dashboard.

Data

Code

The Python package has a somewhat odd structure/organization. I think the main issue is "helper functions" isn't particularly descriptive and contains the bulk of the code. Maybe it could be split up a bit for clarity and/or more manageable expansion as new methods are added (visualzation, filtering, feature extraction, etc.). Just a suggestion.

AJQuinn commented 1 year ago

Thanks @ixjlyons - much appreciated!

We try to avoid detailed discussion on this thread if we can - could you copy the review over to an issue on the source repo and tag this thread in it? You and the authors can discuss any reasonable changes there and report back once things are resolved or if extra input is needed.

Thanks again!

AJQuinn commented 1 year ago

Hi @ajbaird - how are you getting on? do you have an ETA for the review? Thanks for your help!

ajbaird commented 1 year ago

sorry was able to begin the review project and should have it wrapped by the end of the week!

ajbaird commented 1 year ago

Hi All, Here are my thoughts

Overview

The document is well constructed and checks all of the boxes needed for a JOSS publication. Statement of need is clear state of the field and summery are well written. I do wish there was a section on use case, or context of use that could guide a user around API and data sets available.

I found a large lack of clarity around the repository documentation, especially regarding datasets that can be used to test this software, how they can be leveraged and a description of the output targets.

Paper

Documentation

install

The notes here weren't complete for all platforms (non conda environments) and had an issue with tk as a dep. Here are my notes:

Proof of context

I aimed to work my way through the jupyter notebook titled basic_emg_analysis using the zenodo synthetic data set. I had the following issues regarding this process and was not able to generate the figures at the end of the notebook. Here are my notes:

why is the directory information for the data at least not set to some default value, it is very confusing setting up new directories and configure files that restrict even running the most basic example without

Conclusions

Considering these issues in getting a viable proof of context and use, I feel like there are edits and changes needed to the repo documentation in order to approve publication of this paper. Please address to improve documentation.

drcandacemakedamoore commented 1 year ago

Thank you for your thoughtful reviews @ajbaird and @ixjlyons . There are many changes we will work on over the summer. I would note that there is synthetic data at https://github.com/ReSurfEMG/synthetic_data which has been archived on Zenodo. As the data is in .npy format we will better instructions in cases where .npy can not be used directly i.e. where we begin with a Poly5 file. There are many converter functions for different lab groups on the repository in converter_functions.py; but a converter function to a Poly5 is a bit of an unnecessary step backwards. We really appreciate your thoughtful and thorough reviews, and will notify you as we update the library and paper.

Kevin-Mattheus-Moerman commented 1 year ago

@AJQuinn can you please pick this up again? Thanks

drcandacemakedamoore commented 1 year ago

@Kevin-Mattheus-Moerman , @ixjlyons and @ajbaird Thank you all for your thoughtful comments and review. We have now made several changes to the repository. For example in our basic EMG analysis notebook you should now be able to load numpy (the format of our sample data) or Poly5. We made almost all the changes suggested except splitting up helper functions. We felt this part of the project should be worked on with the main users, so it is still ongoing. I believe I have fixed the tkinter issue, and my installation instructions are more clear and inclusive. I would love to hear if this is now working for everyone.

AJQuinn commented 1 year ago

Hi @drcandacemakedamoore - thanks for the response to feedback. I can see lots of changes on the repo but I'm afraid I'm struggling to match them up to bits of specific feedback.

To make the next step simpler, would you mind commenting with a quick summary of specific PRs/commits that relate specifically to the JOSS review? or better still, link this thread into the comments of an active issues/PR. They should then appear here for reference.

@Kevin-Mattheus-Moerman , @ixjlyons and @ajbaird - please can you try and get to this in the next two weeks if at all possible. Please let me know if you'll need more time.

drcandacemakedamoore commented 1 year ago

I have started a table as below (in the next comment there is a link to a full linked version):

Suggestion/Problem Commits PR Number(s) file and area
All requested changes on JOSS paper e9658c24bfb2f3b9c89b09f7d3a8829ce652dd16 247 paper.md
Creating a conda environment ran for a very long time resolving 99ff683a9716761be44a5833616ccb87c44afaf9 246 README.md
Developer setup name confusion efa8ab910e68217c51274a9b306e947025d444b9 260 README.md
Instructions for each architecture supported 902dfc3a7a5e873690a7355d10c212a70f4e3ef8 , 93a5bea7da77fbaa33028cea031028e52dd83e67 253 README.md
No tutorial on how to use synthetic data to run a simulation to verify functionality 7f317a2f5691c3b3ca88535861d0f273ea660ec8 259 researcher_interface/getting_started.ipynb
tkinter not included in dependancies 53eb1f3ab9ad226ff9163e865efe540b9491e56e 250 environment.yaml
Iconsider uploading the sample dataset somewhere it can more easily be retrieved (without Docker), e.g. Dryad, Zenodo, OSF. on Zenodo seperate repo(https://github.com/ReSurfEMG/synthetic_data) DOI
a walkthrough of generating a synthetic dataset sufficient to get started on a notebook or two would be good 7f317a2f5691c3b3ca88535861d0f273ea, older commit to readme for command line 259 by commmand line in README.md, by notebook in researcher_interface/getting_started.ipynb
example json formatting doesn't work across architectures, double quotes are needed 3f09f6f46f58e50c4c41cbf9c18749ec402c716d 253 README.md

Please let me know if this is a good format for you guys Also there is a hot-linked version : (https://github.com/ReSurfEMG/ReSurfEMG/issues/252#issuecomment-1677346866)

drcandacemakedamoore commented 1 year ago

I think I have responded to almost all comments by providing links to the PR and commit number, or explaining our ideas on the issue here: (https://github.com/ReSurfEMG/ReSurfEMG/issues/252#issuecomment-1677346866). We have to work with the main end users of the library at the university medical centers to split up the helper functions module in a way that makes sense, and this will take time.

AJQuinn commented 1 year ago

Amazing, thank you @drcandacemakedamoore - over to @ixjlyons and @ajbaird

ajbaird commented 1 year ago

hi @drcandacemakedamoore thank you so much! I thought that the repo was super duper close to checking all the boxes for me, just with a better readme and instructions on linux installation. I'll be able to look over it in the coming two weeks! Thanks again for all the hard work!

ixjlyons commented 1 year ago

@editorialbot generate pdf

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

ixjlyons commented 1 year ago

We made almost all the changes suggested except splitting up helper functions. We felt this part of the project should be worked on with the main users, so it is still ongoing.

Completely reasonable. It was meant as a suggestion, but I'm happy to hear you're working on it. I think it'll make the code a little easier to start working with, but good choice taking care of existing users.


I've reviewed the changes made and it all looks good to me. I went through the installation instructions and ran through a couple notebooks again to check, so I've checked off everything in my checklist.

AJQuinn commented 1 year ago

Great - thank you @ixjlyons - I'll ask for one further confirmation that you're happy when we're onto the next phase.

@ajbaird - how are you getting on with the review? please let me know if you need anything from our side!

ajbaird commented 1 year ago

working this today, should have it wrapped up end of day! Sorry for the latency.

ajbaird commented 1 year ago

Hi All!

Was just able to pull the repo with all the changes you've commmited and I'm really impressed! you've absolutely hit all the issues I had last time around. Super easy to begin working on the notebooks on a linux platform. I really commend you for all the effort you put into it. Thank you. I know it was a lot of asks and everyone is so busy these days :)

@AJQuinn Paper is good to go from my point of view. Excellent tools, excellent documentation. Really well done!

Thanks again @drcandacemakedamoore this repo will be a treat for a lot of people to play around with!

Austin

drcandacemakedamoore commented 1 year ago

Thank you guys who reviewed ( @ixjlyons and @ajbaird )! Your comments were really useful in terms of making the package more broadly usable, and the user side of the team is and will continue refactoring and adding a bit more code based on some of them. Please let us know if there is anything missing to go to the next steps in the publication process @AJQuinn .

AJQuinn commented 1 year ago

Hi @drcandacemakedamoore - thanks for addressing the reviewer comments, I'll make a start on the next steps!

@ixjlyons - is everything good to go from your perspective? - just need a final 'go' confirmation to wrap things up.

Thanks @ajbaird for the review

ixjlyons commented 1 year ago

@ixjlyons - is everything good to go from your perspective? - just need a final 'go' confirmation to wrap things up.

Yep :+1:

AJQuinn commented 1 year ago

Super - thanks all!

AJQuinn commented 1 year ago

@editorialbot generate pdf

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left: