search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
709 stars 38 forks source link

[REVIEW]: Automated Sleep Stage Scoring Using k-Nearest Neighbors Classifier #2377

Closed whedon closed 4 years ago

whedon commented 4 years ago

Submitting author: @teamPSG (Tamas Kiss) Repository: https://github.com/teamPSG/kNN_Sleep_Scorer_kNNSS Version: v1.0.0 Editor: @oliviaguest Reviewers: @Emma-k-ward, @sbuergers, @Trisbek, @samikakumar Archive: 10.5281/zenodo.4009178

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/5726bdd4b144467c8093e90df0169c25"><img src="https://joss.theoj.org/papers/5726bdd4b144467c8093e90df0169c25/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/5726bdd4b144467c8093e90df0169c25/status.svg)](https://joss.theoj.org/papers/5726bdd4b144467c8093e90df0169c25)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@Emma-k-ward & @sbuergers, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @oliviaguest know.

✨ Please try and complete your review in the next six weeks ✨

Review checklist for @Emma-k-ward

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @sbuergers

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @Trisbek

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @samikakumar

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 4 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @Emma-k-ward, @sbuergers it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

oliviaguest commented 4 years ago

Hey @Emma-k-ward, @sbuergers, and @trisbek this is where the main review occurs, if you need to open something very technical as an issue please do so as an issue at the original repo but all other comments — so most, if not all, of the review — should be here. Hope this is all clear but please ask me any questions you may have! ☺️

oliviaguest commented 4 years ago

@whedon add @Trisbek as reviewer

whedon commented 4 years ago

OK, @Trisbek is now a reviewer

sbuergers commented 4 years ago

Hey @teamPSG,

This is a nicely written and well documented toolbox, well done! I used Matlab 2019b for testing and the SoftwareVerification ran without problems.

From what I could gather there are only a few points to consider:

Contribution and authorship: From the commit history it is unclear who, apart from Tamás Kiss, contributed to the software. Can you include a statement of contributions for the other authors?

References: The references in the main text do not render properly (e.g. [@STEPHENSON2009263,@BASTIANINI2014277...]). It might be that this can simply be fixed by putting a semicolon instead of a comma in-between authors, see this guide.

Community guidelines: I did not see clear guidelines for third-parties wishing to: i.) Contribute to the software, ii.) Report issues or problems with the software (other than contacting the author, but it would be useful to have some pointers for how to use github for this). There are plenty of examples here on github that do this well, for instance here.

@oliviaguest, thanks for setting us up! I take it these comments should not go in the issues of the project?

Cheers, Steffen.

oliviaguest commented 4 years ago

@sbuergers thank you for your feedback so far! These seem to be appropriate to stay here — although feel free to talk to @teamPSG on how to organise these, of course!

teamPSG commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

teamPSG commented 4 years ago

Hi @sbuergers,

Thanks for your feedback and really fast action! Glad you like the toolbox and thanks for the suggestions.

Contributions and authorship: I included a section on who did what at the end of the manuscript.

References: thanks for the hint, it worked, references are now rendered properly (during writing I checked the ms offline using the local compiler and it looked good there -- some components of the compilation workflow might have changed in the course of development...)

Community guidelines: good point, I missed this one. I added a CONTRIBUTING.md to the repo.

I guess the comments are fine right here. Thanks for the review!

Best, Tamás

sbuergers commented 4 years ago

Hi @teamPSG,

thanks for implementing the feedback! This looks good to me now.

Best, Steffen.

oliviaguest commented 4 years ago

Hey @Emma-k-ward and @trisbek can you give me a rough ETA for your reviews? I ask not to hurry you but just to be organised. Thank you. ☺️

teamPSG commented 4 years ago

Hi @oliviaguest, have you received feedback on ETA from @Emma-k-ward and/or @trisbek? (The text editor doesn't auto-complete the mention fortrisbek for me -- does it mean anything?) Thanks!

trisbek commented 4 years ago

I have not been able to run anything on my house PC so I will check with Samika later (she is in californian time) and let you know/ tris

El lun., 6 jul. 2020 a las 9:42, teamPSG (notifications@github.com) escribió:

Hi @oliviaguest https://github.com/oliviaguest, have you received feedback on ETA from @Emma-k-ward https://github.com/Emma-k-ward and/or @trisbek https://github.com/trisbek? (The text editor doesn't auto-complete the mention fortrisbek for me -- does it mean anything?) Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2377#issuecomment-654099261, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQCEF2ANW6UI6AEQT5ITMSTR2GE6NANCNFSM4OETIXNQ .

oliviaguest commented 4 years ago

@trisbek great! Please tag Samika's username so we can give due credit and assign as reviewer if they so wish, thanks.

oliviaguest commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

oliviaguest commented 4 years ago

@teamPSG are you making any edits at the moment for this? ☺️

teamPSG commented 4 years ago

@oliviaguest no, I have not touched the ms since my response to Steffen (17 days ago according to GitHub)

oliviaguest commented 4 years ago

OK, no problem and let us wait for the reviewers to get back to us. If they don't give a vague ETA within a week or so, I'll email them. ☺️

trisbek commented 4 years ago

Samika ran the code using Matlab 2018B on my behalf. The Software Verification scripts run smoothly and quickly. Function Library scripts look very clearly documented.

Functionality/Performance: The paper states the algorithm's potential to stage sleep across species and genetic/drug manipulations for large animal cohorts, and it states the algorithm has been used already on mice, rats, and non-human primates. But the paper (and Github) only include summary results for 6 knockout mice. If it's already available, it would be nice to see the prediction accuracy output for a larger sample size or at least on other species/conditions to confirm the algorithm's versatility.

(Optional) The algorithm has a nice "deflated single model" to account for imbalanced data (e.g., too few REM epochs). Compared to the original single model, this deflated model results in improved REM prediction accuracy but slightly decreased Wake/NREM accuracy. If I am interested in achieving the greatest accuracy for all sleep stages, it's not obvious to me how I would determine which model to use in the end. It might be helpful to have a small script that outputs which model maximizes the true positive rate for most subjects.

trisbek commented 4 years ago

I have added comments but was unable to tick the boxes on read the PDF today

El lun., 13 jul. 2020 a las 11:50, Olivia Guest (notifications@github.com) escribió:

OK, no problem and let us wait for the reviewers to get back to us. If they don't give a vague ETA within a week or so, I'll email them. ☺️

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2377#issuecomment-657487142, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQCEF2BRAJS3S2OZDU6K4KLR3LRINANCNFSM4OETIXNQ .

oliviaguest commented 4 years ago

@whedon re-invite @trisbek as reviewer

whedon commented 4 years ago

The reviewer already has a pending invite.

@trisbek please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

trisbek commented 4 years ago

Invite me again? [image: image.png]

El mar., 14 jul. 2020 a las 14:58, whedon (notifications@github.com) escribió:

The reviewer already has a pending invite.

@trisbek https://github.com/trisbek please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2377#issuecomment-658196185, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQCEF2G4TZ6SEQKVROZ4KADR3RQBRANCNFSM4OETIXNQ .

oliviaguest commented 4 years ago

@trisbek I cannot AFAIK, you have to be logged in and use the link above.

oliviaguest commented 4 years ago

Perhaps another @openjournals/joss-editors can lend me a helping hand, please? I assume that @whedon is correct and all @trisbek needs to do is click that link? ☺️

danielskatz commented 4 years ago

right - clicking the link should work

oliviaguest commented 4 years ago

Also @trisbek (sorry I keep asking!) can you clarify if Samika wishes to be included as a reviewer? How much of this code and paper have they reviewed, and do they want credit attributed to them for their work?

trisbek commented 4 years ago

Hi,

it does not let me accept and it does not let me see the pdf [image: image.png]

Samika said, next time she will upload her scripts and OSF and the lot to a new account of github but this time she is happy for me to copypaste her comments.

El mar., 14 jul. 2020 a las 15:57, Olivia Guest (notifications@github.com) escribió:

Also @trisbek https://github.com/trisbek (sorry I keep asking!) can you clarify if Samika wishes to be included as a reviewer? How much of this code and paper have they reviewed, and do they want credit attributed to them for their work?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2377#issuecomment-658229517, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQCEF2DNX4RHN2HBPXG7NXLR3RW7BANCNFSM4OETIXNQ .

pdebuyl commented 4 years ago

The pdf file is visible without being logged in. @trisbek did you follow https://github.com/openjournals/joss-papers/blob/joss.02377/joss.02377/10.21105.joss.02377.pdf for the pdf?

oliviaguest commented 4 years ago

@trisbek We have discussed this case (the whole editorial board) as this has not happened at JOSS before (that a reviewer copy/pastes another reviewer's words). We have decided that Samika needs to make a GitHub account (if she doesn't have one already) so that she can receive credit (this should take a few minutes only). Samika does not have to write a separate review since she already did, she merely has to tick the boxes at the top of this issue. This should be very quick and easy — we apologise for bringing this up now but we believe it's open and transparent to do the above. We will email her a link to this GitHub issue so she can comment here with her username. Thank you!

samikakumar commented 4 years ago

Hi all - apologies for the confusion! I've reviewed the scripts and paper, and my comments are above in Tristan's message. I'm happy to tick the boxes once I am added. Thanks.

oliviaguest commented 4 years ago

@whedon add @samikakumar as reviewer

whedon commented 4 years ago

OK, @samikakumar is now a reviewer

oliviaguest commented 4 years ago

@samikakumar thank you so much! I apologise again. You have nothing to apologise for yourself. :relaxed:

trisbek commented 4 years ago

Thanks, this is better, credit where it is due.

Tris

El mié., 15 jul. 2020 a las 7:19, Olivia Guest (notifications@github.com) escribió:

@samikakumar https://github.com/samikakumar thank you so much! I apologise again. You have nothing to apologise for yourself. ☺️

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2377#issuecomment-658569312, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQCEF2AMAKETR22ZGHBIYKDR3VC5VANCNFSM4OETIXNQ .

oliviaguest commented 4 years ago

In case people are reading this in the future for inspiration: please be aware that what happened here is both highly anomalous and frowned upon. JOSS expects reviewers to write their own reviews after, e.g., running the code on their own hardware, etc. Delegation of review to a third party, especially when credit is unallocated, is not inline with our principles and we will be soon updating our guidelines to make this as clear as possible. This is merely being underlined here to avoid setting a bad precedent.

oliviaguest commented 4 years ago

OK, so let's get this back on track! 😅

@teamPSG is the feedback provided by @samikakumar clear in how it relates to possible improvements to your code/paper?

trisbek commented 4 years ago

A key decision and feature to take into account by PSG is to decide if this is good enough for classic EEG human data. For this package to be useful to the thousands of researchers working in EEG human sleep data it would have to be validated in human datasets, beyond the datasets in animals already shown in the pdf and examples. A good dataset is the publicly available https://sleeptight.isr.uc.pt/?page_id=48 and also the recently made available consortium on the subject https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4835314/ Would the authors care to comment how their package links and complements to this wide open initiative? It could make a big difference. This is a extent of influence comment, the documentation is clear, the software runs well.

trisbek commented 4 years ago

Sure, but it was also open, I told you I would ask Samika, she accepted and I told you that she had written the bit I posted a couple of days ago. After your message she came to github and did it herself (good!). It is better this way as it assigns credit where it is due. Now we have the right type of format for my review and a separate one Samika's, as we agreed a few weeks ago.

Hopefully both reviews are useful to @teamPSG in terms of putting their tools in context to the eeg sleep community.

El mié., 15 jul. 2020 a las 10:49, Olivia Guest (notifications@github.com) escribió:

In case people are reading this in the future for inspiration: please be aware that what happened here is both highly anomalous and frowned upon. JOSS expects reviewers to write their own reviews after, e.g., running the code on their own hardware, etc. Delegation of review to a third party, especially when credit is unallocated, is not inline with our principles and we will be soon updating our guidelines to make this as clear as possible. This is merely being underlined here to avoid setting a bad precedent.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2377#issuecomment-658668714, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQCEF2CUPYNGO4SPFIXDSNDR3V3TVANCNFSM4OETIXNQ .

oliviaguest commented 4 years ago

OK, I really feel the need to add one last clarification for future readers, the events are:

@trisbek decided he would ask @samikakumar and did without checking the exact details with me and if I am OK with that. This happened in private mostly. I am, and was, though OK with that, but I want(ed) to do this properly (i.e., give her credit). Therefore, I asked quite a few times of Tristan to tell Samika to join this review with her GitHub account. For whatever reason she didn't. Finally, Tristan explained that he copy-pasted (some of?) her words in his review. So ultimately, I think it's both fair, open, and ethical that Samika takes credit where she did work to improve this paper and code. This is what I asked for from the start (both in private and in this issue thread), as soon as I found out that she had been involved in this process: to include Samika officially, openly, and transparently.

Hope this helps to give the full backstory to readers.

Also thank you for apologising, @trisbek. Please however, let us continue with the actual matter at hand: reviewing this JOSS submission. 😁

Emma-k-ward commented 4 years ago

Hi Olivia, authors,

This is a nice toolbox that was easy to use and gave me easily interpretable output with minimal time investment, even as a totally naive user, which I think was the authors' goal. Some comments on my experience and how I would think it could be slightly improved:

Code I also ran the Software_Verification without issues when running code inside the functions, although a very minor problem came up: when I wrote a main script to call the individual functions it was hard to figure out where to set my working directory. I would usually write my own main script when downloading a package, set the directory and add any paths, and then call functions there, and this was a bit harder here because the directories are called in reference to their values in setup and I wasn't sure how to organise my own script then.

Once I solved that, I could run all the code and everything behaved as expected. I liked that I get output including the structures with original sleep stage labels (I assume this means the manually coded labels for the training) and predicted labels at the end, and the figures are nice. As a naive user I could get interpretable output very quickly, which I think was the goal of the toolbox.

In the code itself I miss some comments to make it clearer what is happening where. The comments that are there are nice, but very high level so I'd like to understand at first glance a little more of what's being calculated. For example, in FirstStep, we get quickly to: %% Calculate features files = dir([par.DataDir filesep '*' par.DataExt]); for file = files' filename = strrep(file.name, par.DataExt, ''); fprintf('Processing %s.\n', filename); generate_statespace(par.TopDir, filename, 'Description', filename, ... 'IsTrainingSet', true, ... 'OutDir', par.IntDir, 'WriteTmpFile', [filename par.PostFn]); end

and inside generate_statespace a few comments say "[see notes inside]", but I don't see much in the way of notes.

Documentation In the documentation, I do see a statement of need and a summary, but I would have liked to know a little bit more. I think the argument is made very clearly that this process should be automated, and I appreciated hearing how the scoring is done manually. However, I don't think the paper is that clear about the advantages of this particular algorithm.

For example, the paper explains that this algorithm is preferable to the previously available algorithms because the features used are human-interpretable but I would've liked one more sentence about this. I only know from the paper that the features are based on power in relevant frequency bands and Hjorth parameters but I don't know in what sense a feature based on those characteristics is more human-interpretable. Are the features literally human-readable? For a single animal would I then be able to dig into the output of generate_statespace and see, as a very simplified example, that a sudden decrease in alpha and increase in delta power are key markers of a certain stage? If so, then I would emphasise this more because I think it's very interesting, but if not I don't understand the type of feature interpretability the authors mean. I also found the Wikipedia citation odd, as I was expecting information about the interpretability of these specific features and instead was led to an article arguing for interpretability in general. Not a problem per se but rather vague. This is rather an important point in the philosophy behind the toolbox and I was also very interested, so I was surprised it was quickly brushed aside.

Then the point that the current method is more valid with animals after drug treatment, or humans with atypical sleep signatures, mostly rests on the argument that the transitions between states are less predictable in these groups, and therefore the current method drops any prior assumptions about transition rates and timing. The authors even state that there's potentially more sleep stages in these atypical cases, which I wouldn't have known and is interesting, although the current method only allows for 3 stages (wakeful, REM, non-REM) so it's less relevant. I would've expected either a reference here (fourth sentence before the heading Brief Software Description) or another sentence saying why it is a reasonable to have such a (lack of) assumption about transition rates. Or, alternatively, that in the output there was a metric showing how often the classifier transitioned between labels so researchers could assess whether they find that reasonable. I think this very small adjustment might be useful to people who want to evaluate whether they will switch to this method.

Overall these are small adjustments and I think it's a cool toolbox!

Best wishes,

Emma

oliviaguest commented 4 years ago

@teamPSG have their work cut out! 💯

Thank you for your detailed feedback, @Emma-k-ward!

teamPSG commented 4 years ago

Thanks for your very good comments and suggestions @trisbek @samikakumar @Emma-k-ward . I'm checking with coauthors and working on the code and the paper.

oliviaguest commented 4 years ago

@teamPSG just to keep track of stuff, when you make changes can you link to them from here (any way you want, link to line numbers, files, PRs, whatever makes sense for you) so we can all see them easily? Also does any of the above by @samikakumar @trisbek @Emma-k-ward make sense to be transferred into a direct code-related issue? If so, please do that to keep things neat and link back to it from here. 😊

teamPSG commented 4 years ago

Hey @oliviaguest, I'll try my best. Is there a previous review that you'd suggest to look at as a good example?

oliviaguest commented 4 years ago

Sure, @teamPSG so here has reviewers opening issues: https://github.com/openjournals/joss-reviews/issues/2285

oliviaguest commented 4 years ago

Here too: https://github.com/openjournals/joss-reviews/issues/2348

teamPSG commented 4 years ago

Hey @oliviaguest, I have a quick technical question regarding editing the paper: I'd like to indicate that some authors have changed their affiliations since the work was done. Like my affiliation should be Pfizer but my current affiliation is Wigner RCP. Is there a specific way to do it or as a workaround shall I just add "Current affiliation" in the name tag in the authors list?