[REVIEW]: SleepECG: a Python package for sleep staging based on heart rate

[editorialbot]

Submitting author: !--author-handle-->@cbrnr<!--end-author-handle-- (Clemens Brunner) Repository: https://github.com/cbrnr/sleepecg Branch with paper.md (empty if default branch): joss Version: v0.5.5 Editor: !--editor-->@emdupre<!--end-editor-- Reviewers: @tompollard, @richrobe, @peterakirk Archive: 10.5281/zenodo.7993717

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/842b3838fa2f0f4f735cbd25de8508bb"><img src="https://joss.theoj.org/papers/842b3838fa2f0f4f735cbd25de8508bb/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/842b3838fa2f0f4f735cbd25de8508bb/status.svg)](https://joss.theoj.org/papers/842b3838fa2f0f4f735cbd25de8508bb)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@tompollard & @richrobe & @peterakirk, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @emdupre know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @richrobe

📝 Checklist for @peterakirk

📝 Checklist for @tompollard

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1111/j.1469-8986.1966.tb02650.x is OK
- 10.1109/TBME.1985.325532 is OK
- 10.1161/01.CIR.93.5.1043 is OK
- 10.5664/jcsm.26814 is OK
- 10.3389/fpubh.2017.00258 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.13 s (568.0 files/s, 191457.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
SVG                              8              0            109          17772
Python                          32            901           1611           3377
Markdown                        19            189              0            601
C                                1             67            134            360
YAML                             8             40              4            280
HTML                             3              0              0             55
TeX                              1              5              0             48
TOML                             1             12              0             24
JavaScript                       1              2              0             15
CSS                              2              1              0             10
-------------------------------------------------------------------------------
SUM:                            76           1217           1858          22542
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 720

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[emdupre]

👋 Hi @tompollard, @richrobe, and @peterakirk, and thank you again for agreeing to review this submission !

The review will take place in this issue, and you can generate your individual reviewer checklists by asking editorialbot directly with @editorialbot generate my checklist.

In working through the checklist, you're likely to have specific feedback on SleepECG. Whenever possible, please open relevant issues on the linked software repository (and cross-link them with this issue) rather than discussing them here. This helps to make sure that feedback is translated into actionable items to improve the software !

If you aren't sure how to get started, please see the Reviewing for JOSS guide -- and, of course, feel free to ping me with any questions !

[cbrnr]

Looking forward to your comments and reviews! Pinging @hofaflo, who implemented the bulk of SleepECG (and who is therefore of course also an author of the submitted paper)!

[richrobe]

Review checklist for @richrobe

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/cbrnr/sleepecg?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@cbrnr) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[emdupre]

@tompollard and @peterakirk, I just wanted to check-in as I noticed that you haven't yet generated your reviewer checklists. Please let me know if you're having any trouble accessing these !

Happy Thursday everyone, and May the Fourth be with you 😄

[peterakirk]

Review checklist for @peterakirk

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/cbrnr/sleepecg?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@cbrnr) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[emdupre]

:wave: Hi @peterakirk ! I notice that you have completed your checklist but have not opened any issues on the software repository. Is there another place that you're providing feedback to the authors ? Please let me know !

[peterakirk]

Hi @emdupre, apologies, I've gone through the checklist but had other stuff come up, so had to put it on pause for the moment. As of yet, I have had no issues: everything has installed fine, runs smoothly, and documentation looks good. Just remaining is I'll make sure all the basic metrics look appropriate, which I'll try and get to next week.

[richrobe]

Hi @cbrnr,

thank you for putting together such a lovely package! In general, I like it very much and see it as a valuable contribution to sleep staging estimation from ECG data. After having gone through code, documentation and paper I'll summarize my thoughts below and have created extra issues in the package repository.

General Remarks

• In the README, you wrote: "Based only on ECG (and to a lesser extent also movement data), SleepECG provides functions for [...]". I'm wondering where the movement data comes from since you didn't mention it elsewhere and also have no functions for importing, processing, and extracting features from acceleration or actigraphy data.

See also

• https://github.com/cbrnr/sleepecg/issues/155

Paper

The paper is easy written, easy to follow, and nicely outlines the need of the SleepECG package. It provides a good motivation, especially regarding the reproducibility of the existing approaches. Just some minor comments:

• To better understand the scope of the package at first sight, an overview image showing the typical workflow for ECG-based sleep staging (data download, data cleaning, pre-processing, feature extraction, training, ...), and where SleepECG could help in this process (namely everywhere) would be beneficial.
• Could you add some more information on the pre-trained classifiers you used in your paper in terms of network architecture, training & evaluation process, etc.? Is there already a publication on it (if yes, could you link it?)

See also

• https://github.com/cbrnr/sleepecg/issues/156

Installation

• The straightforward installation of SleepECG fails on my system (Python 3.9 on 2017 Macbook Pro with macOS Ventura 13.3, Intel CPU, no GPU) using the current pyproject.toml version with the following error message: with the following error message:

  Could not find a version that satisfies the requirement tensorflow-macos>=2.7.0; (sys_platform == "darwin" and python_version < "3.11") and extra == "full" (from sleepecg[full]) (from versions: none)

  However, replacing the optional dependencies tensorflow-macos and tensorflow-metal by tensorflow in the pyproject.toml file fixes the issue and correctly installs the package with all dependencies.

• Additionally, when installing SleepECG with extras, following the instructions in the Contributing guide dev dependencies fails when using zsh as shell (which is now the default unser ) because square brackets need to be escaped. Thus, I would maybe suggest to add an additional note that zsh users should call pip install -e ".[dev]" (with quotes). This command should also work when using other shells (such as bash).

See also

• https://github.com/cbrnr/sleepecg/issues/157
• https://github.com/cbrnr/sleepecg/issues/158

Examples

• Example try_ws_gru_mesa.py: hypnogram isn't shown when running the code because plt.show() is missing in the end. Adding this line correctly displays the figure.

See also

• https://github.com/cbrnr/sleepecg/issues/159

Other than that, I have nothing to add. Let me know what you think of my suggestions! 🙂

[tompollard]

emdupre I am looking at this today! Sorry it's taken me a while to get to it.

[tompollard]

Review checklist for @tompollard

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/cbrnr/sleepecg?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@cbrnr) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[cbrnr]

@richrobe thank you so much for your detailed comments! I've already started addressing them in the issues you created!

[tompollard]

Hi @cbrnr -

I agree with @richrobe. sleepecg is a helpful, well-described, easy to use software package. I added some thoughts below, but I also think that the software is in great shape as it stands. My comments mostly relate to ongoing maintenance tasks rather than things that need to be dealt with right now.

While I looked over the code (which is cleaner than 99% of code I come across, and certainly all of mine!) I didn't do a deep dive into any of the algorithms. As the package provides a framework for supporting a breadth of algorithms and does not claim to implement successful, validated sleep staging (at least right now), this didn't seem especially necessary.

Installation

Dependencies

For development I'd usually expect to see a file containing the dependencies somewhere (e.g. requirements.txt or environment.yml). Does this exist? I see that the dependencies are listed in README.md, but it would be nice to have a file that can be used by package managers (e.g. pip, conda).

pip

pip install sleepecg correctly installs sleepecg-0.5.4 on my machine (Apple M1 Max, macOS Ventura 13.3, Python 3.10, pip 23.0).

As mentioned by @richrobe in https://github.com/cbrnr/sleepecg/issues/158, pip install sleepecg[full] displays a no matches found: sleepecg[full] error in zshell.

pip install 'sleepecg[full]' correctly installs sleepecg-0.5.4 (full) on my machine.

import

When importing sleepecg (v0.5.4), I see the following deprecation notices:

>>> import sleepecg
/Users/tompollard/sand/sleepecg/env/lib/python3.10/site-packages/sleepecg/heartbeats.py:638: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  _squared_moving_integration_numba = jit(_squared_moving_integration_py)
/Users/tompollard/sand/sleepecg/env/lib/python3.10/site-packages/sleepecg/heartbeats.py:639: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  _thresholding_numba = jit(_thresholding_py)

It looks like this is already addressed in the repo, so I haven't raised an issue: https://github.com/cbrnr/sleepecg/blob/d3d15d5bfa6386bdc68fb3c9bfc58d8da5f50d01/sleepecg/heartbeats.py#L638-L640

Code

Relative vs absolute imports

Absolute imports seem to be generally preferred by Python developers (https://peps.python.org/pep-0008/#imports). sleepecg uses relative imports in places (e.g. from .physionet import _list_physionet). Is this intentional? If not, perhaps switch to absolute imports at some point?

Scripts

The scripts (e.g. examples/benchmark/plot_benchmark_results.py) are missing structure that I think can be helpful (e.g. if __name__ == "__main__":). Minor thing and not necessary, but might be worth adding some of this structure (e.g. to prevent them running when imported)? This is the outline I use: https://www.annasyme.com/docs/python_structure.html.

Command line interface

Adding a command line interface for the package via pyproject.toml might be a nice addition, though not necessary.

scipy.misc.electrocardiogram has been deprecated

The example code at https://sleepecg.readthedocs.io/en/stable/heartbeat_detection/#usage uses scipy.misc.electrocardiogram which has been deprecated. Needs updating to scipy.datasets.electrocardiogram at some point.

ecg = electrocardiogram()

<stdin>:1: DeprecationWarning: scipy.misc.electrocardiogram has been deprecated in SciPy v1.10.0; and will be completely removed in SciPy v1.12.0. Dataset methods have moved into the scipy.datasets module. Use scipy.datasets.electrocardiogram instead.

Paper

Comments on the paper at: https://github.com/openjournals/joss-papers/blob/joss.05411/joss.05411/10.21105.joss.05411.pdf

Figure 1

I'm not sure I understand the Y-axis labels on Figure 1. Could you add details to either the axis label or the caption? The caption of axis label should highlight the log scale. LTDB should be defined somewhere in the text or caption somewhere. The algorithms should also be defined (e.g. it is unclear to me what "mne" means).

Add heartbeat detection metrics figure?

There is a metrics figure at https://sleepecg.readthedocs.io/en/stable/heartbeat_detection/#usage that I think would be a nice addition to the paper.

Add sleep staging figure?

Given the focus of the package is sleep staging, I think it would be nice to add a figure demonstrating the sleep staging functionality. A (ideally non-cheat!) version of the hypnogram at https://sleepecg.readthedocs.io/en/stable/plot/ might be a good one to include?

State of field

I'm not familiar with the literature on sleep-staging with ECG, but I do see there are one or two open source codebases that claim to enable ecg-based sleep staging. I wonder whether these should be briefly mentioned?

e.g. should comparisons be made to https://github.com/bdsp-core/ecg_respiration_sleep_staging? The repo doesn't look completely baked (e.g. "training_code_still_messy)", so perhaps it reinforces your point that "either the processing pipeline or the dataset used to generate the published results are not publicly available."

Reiterate point that sleep staging is not a solved problem?

I think it would be good to reiterate the point that sleep staging is not a solved problem. This is mentioned in the paper, but I think it is helpful for the user to be reminded that the package is not claiming (currently) to be able to successfully stage sleep with ECG.

References

Not completely necessary, but ideally I think the algorithms and datasets should be formally cited in the paper if they are being discussed. This helps to credit the creators and also supports the idea of reproducibility.

Archival version of the code

It wasn't clear to me whether there is an archival version of the software anywhere. If not, it would be great if could archive a copy (with DOI) somewhere like Zenodo or Figshare. I have a conflict of interest, but we would also be happy to host the software on PhysioNet.

Documentation

import numpy as np should be moved to the code block above at https://sleepecg.readthedocs.io/en/stable/heartbeat_detection/#usage (to avoid rri = 1000 * np.diff(beats) / fs failing with an import error).

[cbrnr]

Thank you @tompollard, these are great and thoughtful comments! It will take me some time to process and address each of them, and I'll be using this thread here and/or the issues you opened!

[cbrnr]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[cbrnr]

@tompollard I've addressed most of your comments in the corresponding issues, so here I'm only going to reply to points where this is not the case (but I'll indicate the issues where relevant).

• [x] Dependencies: See https://github.com/cbrnr/sleepecg/issues/163.
• [x] pip install in zsh: Fixed in https://github.com/cbrnr/sleepecg/pull/160.
• [x] Numba deprecation warning: Fixed in https://github.com/cbrnr/sleepecg/pull/151.
• [x] Relative vs. absolute imports: Fixed in https://github.com/cbrnr/sleepecg/pull/170.
• [x] Structure in scripts: See https://github.com/cbrnr/sleepecg/issues/165.
• [x] CLI: Currently I don't see the immediate benefit of adding a CLI, but that's maybe something to keep in mind.
• [x] scipy.misc.electrocardiogram: See https://github.com/cbrnr/sleepecg/issues/166.
• [x] Figure 1: The y-axis label reads "Mean runtime (s)", which is an average over multiple datasets. I'm not sure it is worth including all the details about how we ran the benchmarks, but I could refer to the description of the benchmark script. I'll also add something like "note the logarithmic scale of the y-axis" to the caption to draw attention to that (important) fact. I'll spell out LTDB. I'll think about how to best describe the different detectors (such as mne), because we might be approaching the word limit in this paper. But you are absolutely right that something needs to be added to clarify!
• [x] Adding more figures: See https://github.com/cbrnr/sleepecg/issues/168.
• [x] State of field: See https://github.com/cbrnr/sleepecg/issues/169.
• [x] Reiterate: Agreed, I'll add a sentence near the end of the document!
• [x] Citing algorithms and data sources: We've cited the Pan & Tompkins algorithm, which other algorithms are you referring to? Currently, we've linked the data sources instead of citing them, because I'm not sure that every database has its own publication (but I could take a look of course).
• [x] Code archive: A Zenodo DOI will be created as part of the JOSS publication (once and if it gets accepted).
• [x] Documentation: Fixed in https://github.com/cbrnr/sleepecg/pull/171.

[tompollard]

@cbrnr thanks for your quick response. @emdupre as far as I'm concerned, the code and paper look great and they are fine to publish in their current state!

[cbrnr]

Thanks @tompollard, I do want to include pointers to related repos (I need to check them out first) and a hypnogram. I'll report back once this is done.

[cbrnr]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[cbrnr]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[cbrnr]

I think I've addressed all comments. The latest paper version is linked above. Let me know if there is anything else!

[peterakirk]

Hi all,

Firstly, well done to @cbrnr and @hofaflo for curating what appears to be a great package. I’m not as familiar with the sleep literature, but classifying sleep stages from ECG is not an approach I had heard of before. This package seems like a great way to make this accessible to the field, in a reproducible and open manner.

I’ve not too much to add. I agree with the other reviewer comments, the most of which seem to have already been addressed by the authors. Below are just a few quick comments (the latter of which I've opened as issues):

• Everything installed smoothly for me (PyCharm 2022.2.2 CE IDE, Python 3.9, Precision 5820, Windows 10, Intel i9).
• I ran a quick sanity check for HRV metrics on an open ECG dataset (https://doi.org/10.13026/sq6q-zg04). RMSSD values appear to correlate highly (ICC=.94, p < .0001) between SleepECG and our package, RapidHRV, after accounting for the lack of unit normalization (see next comment).

• Unless I have misunderstood, HRV metrics (e.g., RMSSD) do not appear to be in normalized units. If I saw RMSSD values, I would typically expect them to be in ms units (in the 10-100 range). This should be explained in the documentation or changed accordingly.
• It would be useful to include a brief example usage on the github readme.md.

[cbrnr]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[peterakirk]

@emdupre, the authors have addressed my comments and everything looks good to me.

[emdupre]

🙏 Thank you @peterakirk, @tompollard, and @richrobe for your thoughtful reviews -- and @cbrnr for your engagement throughout this process !

I'll perform a few editorial checks now and update on those shortly.

[emdupre]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1111/j.1469-8986.1966.tb02650.x is OK
- 10.1109/TBME.1985.325532 is OK
- 10.1161/01.CIR.93.5.1043 is OK
- 10.5664/jcsm.26814 is OK
- 10.3389/fpubh.2017.00258 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[emdupre]

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

• [x] Double check authors and affiliations (including ORCIDs)
• [x] Make a release of the software with the latest changes from the review and post the version number here. This is the version that will be used in the JOSS paper.
• [x] Archive the release on Zenodo/figshare/etc and post the DOI here.
• [x] Make sure that the title and author list (including ORCIDs) in the archive match those in the JOSS paper.
• [x] Make sure that the license listed for the archive is the same as the software license.

Editor Tasks Prior to Acceptance

• [x] Read the text of the paper and offer comments/corrections (as either a list or a PR)
• [x] Check the references in the paper for corrections (e.g. capitalization)
• [x] Check that the archive title, author list, version tag, and the license are correct
• [x] Set archive DOI with @editorialbot set <DOI here> as archive
• [x] Set version with @editorialbot set <version here> as version
• [x] Double check rendering of paper with @editorialbot generate pdf
• [x] Specifically check the references with @editorialbot check references and ask author(s) to update as needed
• [x] Recommend acceptance with @editorialbot recommend-accept

[cbrnr]

• [x] Double check authors and affiliations (including ORCIDs)
• [x] Make a release of the software with the latest changes from the review and post the version number here. This is the version that will be used in the JOSS paper.
• [x] Archive the release on Zenodo/figshare/etc and post the DOI here.
• [x] Make sure that the title and author list (including ORCIDs) in the archive match those in the JOSS paper.
• [x] Make sure that the license listed for the archive is the same as the software license.

The version number to be used in the JOSS paper is v0.5.5. The Zenodo DOI is 10.5281/zenodo.7993717.

[emdupre]

Thank you, @cbrnr !

I've finished reviewing the code and software and am also very happy with the state of the submission. I did have a few editorial requests :

• Overall, please add citations when possible for the described datasets; e.g., MESA, SHHS.
• In the first footnote, the github.com/bzhai/multimodal_sleep_stage_benchmark codebase has requested that they be acknowledged with a citation. Could you please update to include and cite that reference in the footnote ?
• On L33 and L34, please add abbreviations for MESA and SHHS, as these are subsequently referenced but not explicitly defined.
• On L46, please change to described in @TFESC:1996 and @Shaffer:2017. to correct in-text formatting
• On L52, please add a citation for Tensorflow (I would recommend using the Zenodo record; i.e., https://zenodo.org/record/7987192)
• On references specifically, I noticed that all of your title fields are in double-brackets and so following the exact capitalization that you include in your .bib file. Is there a reason some of these are in sentence case while others are not ? It would be helpful to standardize as possible, either by removing one set of brackets or by explicitly confirming the casing of all titles.

[emdupre]

@editorialbot set v0.5.5 as version

[editorialbot]

Done! version is now v0.5.5

[emdupre]

@editorialbot set 10.5281/zenodo.7993717 as archive

[editorialbot]

Done! archive is now 10.5281/zenodo.7993717

[cbrnr]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[cbrnr]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[cbrnr]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[cbrnr]

@emdupre done. I've used {{ }} for titles to ensure that the original casing is used, but I've found and corrected one error. Now all titles should be consistent and correct.

[cbrnr]

@editorialbot generate pdf