search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
725 stars 38 forks source link

[REVIEW]: BCImat: a Matlab-based framework for Intracortical Brain-Computer Interfaces and their simulation with an artificial spiking neural network #3956

Closed whedon closed 2 years ago

whedon commented 2 years ago

Submitting author: !--author-handle-->@eferrea<!--end-author-handle-- (Enrico Ferrea) Repository: https://github.com/eferrea/BCI-Master Branch with paper.md (empty if default branch): Version: v.1.0.0 Editor: !--editor-->@oliviaguest<!--end-editor-- Reviewers: @mstimberg, @puolival Archive: 10.5281/zenodo.6759182

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/b6f403bdc8f1b9c96698feca6a466169"><img src="https://joss.theoj.org/papers/b6f403bdc8f1b9c96698feca6a466169/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/b6f403bdc8f1b9c96698feca6a466169/status.svg)](https://joss.theoj.org/papers/b6f403bdc8f1b9c96698feca6a466169)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@mstimberg & @puolival, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @oliviaguest know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @mstimberg

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @puolival

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 2 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @mstimberg, @puolival it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 2 years ago

Wordcount for paper.md is 1175

whedon commented 2 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.09 s (332.1 files/s, 36526.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
MATLAB                          16            371            342           1071
C++                              4            216            121            488
XML                              3              0              0            176
TeX                              1             13              0            173
Markdown                         2             53              0             95
C/C++ Header                     1              9             10             20
YAML                             1              1              4             18
DOS Batch                        1              0              0              9
-------------------------------------------------------------------------------
SUM:                            29            663            477           2050
-------------------------------------------------------------------------------

Statistical information for the repository '72453a089fb461b9e3bf5986' was
gathered on 2021/11/29.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
nerofumo82                      10           918             54          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
nerofumo82                  864           94.1          0.1               15.39
whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1126/science.aaa5417 is OK
- 10.1016/S0140-6736(17)30601-3 is OK
- 10.1016/S0140-6736(12)61816-9 is OK
- 10.1038/nn.3265 is OK
- 10.1038/nature11076 is OK
- 10.1073/pnas.0808113105 is OK
- 10.1038/nature10845 is OK
- 10.1038/nature13665 is OK
- 10.1016/j.neuron.2014.08.038 is OK

MISSING DOIs

- 10.1101/2021.02.16.431440 may be a valid DOI for title: Statistical determinants of visuomotor adaptation in a virtual reality three-dimensional environment
- 10.1523/jneurosci.02-11-01527.1982 may be a valid DOI for title: On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex
- 10.1109/thms.2020.2968411 may be a valid DOI for title: Brain–computer interface software: A review and discussion

INVALID DOIs

- None
whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

eferrea commented 2 years ago

Hi @oliviaguest, I am here in the new page. Thanks @puolival for reviewing the paper.

oliviaguest commented 2 years ago

Hi all! 👋 Thank you so much @mstimberg, @puolival for accepting to review this. Please read the instructions above. Any questions, feedback on the paper, etc., please post here. Any very code-specific questions, suggestions, etc., please use the issues in the code repo and link to them from this thread, so we can all keep track of them. 🌸

For examples of how this process plays out feel free to skim previous reviews, such as: #2285 and #2348. ☺️

mstimberg commented 2 years ago

Review @mstimberg

As mentioned earlier, I am neither a MATLAB-expert, nor do I know much about BCI – so please read my review with this in mind.

I was able to install and compile all dependencies, as well the TrackM software and the mex files for the Matlab GUI presented in this paper (under Ubuntu Linux 20.04). I was able to run both pieces of software and test its basic functionality, and did not encounter any major problem. I do not have access to Blackrock neural interface, so I obviously could not test the package in a real BCI application. I have opened a number of issues on the main repository (eferrea/BCI-Master#2, eferrea/BCI-Master#3, eferrea/BCI-Master#4, eferrea/BCI-Master#5, eferrea/BCI-Master#6), with suggestions for improvements to documentation, usability and testing. I fear that in its current state, the package will be difficult to use for new users, and some parts of it look like code that is rather aimed for internal than for general use (e.g. hardcoded private IP addresses). Everything looks to be fixable, though, I did not encounter any major blocking issues. Regarding the Software paper, I do not have much to criticize. As friendly whedon noted earlier, three papers are missing DOIs and one of them is also missing the year (leading to a "n.d." in the citation).

eferrea commented 2 years ago

Hi @mstimberg, many thanks for the constructive review. I will address all of your concerns as soon as possible.

whedon commented 2 years ago

:wave: @mstimberg, please update us on how your review is going (this is an automated reminder).

whedon commented 2 years ago

:wave: @puolival, please update us on how your review is going (this is an automated reminder).

eferrea commented 2 years ago

Hi @mstimberg and hi @oliviaguest , I implemented all suggestions to improve documentation and code usability. You can check my answers directly on the open issues. I also updated the missing DOI and the missing year. Thanks.

mstimberg commented 2 years ago

@eferrea thanks for the substantial changes, all of my concerns have been addressed (except for the few minor issues I commented in eferrea/BCI-Master#2, eferrea/BCI-Master#3, eferrea/BCI-Master#4). Before the formal acceptance from my side, there's only one more thing that I unfortunately forgot to mention earlier: it would be good to make an official release on github (or just a tag) and give the software a version number. This will make it much clearer which version of the code was described by the JOSS paper, and will allow you to easily communicate changes with respect to this version in the future. But it might make sense to wait with this until the end of the reviewing process, of course.

mstimberg commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

eferrea commented 2 years ago

@mstimberg , thanks for your comments. I answered your questions (eferrea/BCI-Master#2, eferrea/BCI-Master#3, eferrea/BCI-Master#4). In addition, I have to mention that I just added in the paper.md a small paragraph of our funding sources. As you suggested, I will then wait until the end of the review process to make an official release.

mstimberg commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

mstimberg commented 2 years ago

Thanks for the additional changes @eferrea. I just noticed that the paper still has a broken reference ([RN1] in line 76), but apart from that I am happy with the current state of the software/paper.

eferrea commented 2 years ago

Many thanks @mstimberg for your review. I also corrected the broken reference now.

mstimberg commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

mstimberg commented 2 years ago

Great, so just to make it official: I recommend acceptance (CC @oliviaguest )

oliviaguest commented 2 years ago

@mstimberg thank you so much!

@puolival is everything clear with what you need to do to review this? 😊

puolival commented 2 years ago

@oliviaguest I am almost finished writing the review. I'll report back in a couple of days.

puolival commented 2 years ago

Review 21.1.2022

Manuscript

  1. The manuscript is currently too short. Please report results at least from one actual experiment. How good cursor control performance can be achieved using this software? Please also compare the performance to a typical state-of-the-art BCI experiment. Please also state more clearly the main functions of this software and how does it compare to the other existing software. Please also mention the most important limitations and/or how the software can be developed further.

  2. It is unclear what is meant by the sentence “The code was successfully used from extracted neural activity of rhesus monkeys performing a similar center-out reach as previously described in a virtual-reality environment …”. If this means that experiments on rhesus monkeys were performed for the purpose of developing this software, the relevant ethical permissions, procedures, methods, etc. must be carefully reported according to standard good practices. It is not possible to recommend publication if an animal model was indeed used and these information are missing.

  3. Related to points 1. and 2., is this software mainly intended to be used with recordings done on humans? This is currently unclear due to the reference to animal models. The abstract only refers to human patients. If the software is intended to be used with human patients, it must be tested with experiments involving human patients.

  4. Does the statement “… movement stage information and position of the target are handled to clear the spike buffer regularly to avoid excessive memory overload” mean that the size of the buffer is dynamic during run-time? It would be much more preferable to use a constant-sized circular buffer (also known as circular queue or a ring buffer) instead, which would not have such issues.

  5. Has this software been tested with any other acquisition system than Cereplex? The Cereplex system is now mentioned as an example but if the software is not (known to be) compatible with other acquisition systems, it should be clearly stated. The manuscript now gives an impression of a general-purpose system but if the software is in fact written for a specific hardware vendor, excluding the SNN part, this should be clearly communicated, perhaps already in the title.

  6. “We also implemented some important BCI features that are frequently used in the literature to provide efficient training and better decoding performance”. How much did the performance improve? How was the performance estimated? Please report the results that were obtained.

  7. Please proofread the manuscript once more. In particular, please check that words that should be in singular form are not accidentally in plural form or vice versa. Spaces are sometimes missing before citation parentheses. Check that words are capitalized correctly (e.g. MATLAB is sometimes referred to as ‘Matlab’ and sometimes as ‘MATLAB’).

  8. In the reference list, journal names are sometimes written in full and sometimes abbreviated. Please adhere to a consistent style.

  9. Font sizes in Figure 1 are too small. Please ensure that all text is readable when the manuscript is printed.

  10. Please write all abbreviations once in full.

  11. To make the manuscript more accessible, it would be good if it was explained what is a “standard c++ client-server application”.

  12. The main features of the SNNs must be described (which neuron model is used, size and connectivity of the network, is the network static or is there plasticity, etc.)

Software

  1. The license file contains the name of only one author while several authors have written the software (based on comments in the source files).

  2. There is no automated package management solution, which is a JOSS recommendation. Also, the installation instructions do not contain version numbers of external packages (e.g. VRPN), which is likely to cause issues sooner or later. The user also needs to manually copy files in order to get the software running, which is a step that I think should be automated.

  3. There are no automated tests. The files “TestBlackrock.m” and “TestVRPNConnection.m” (neither of which is properly commented) can be used to test hardware/internet connections but there are no tests for assessing functionality of this software. The authors should write at least unit tests for all functions.

  4. Many functions and methods are very long, might do several things, and are often not easy to understand. I suggest that long functions are refactored into several smaller ones, such that each function does one only thing (i.e. adherence to the so-called keep-it-simple principle).

  5. There are comments such as “… many of the parameters listed below are not used with the basic provided task controller but were used during our experiments”. It therefore appears that this software might not yet be suitable for general use, since it has not been written in a general-purpose context.

  6. All functions, their input/output arguments, and non-trivial block of code should be properly commented. Indentation should be uniform across files. Single-letter variable names should not be used unless they are part of a mathematical equation.

  7. Adhering to a specific style guide (see e.g. https://google.github.io/styleguide/) would make it easier to read the program code.

  8. The software would be more valuable for researchers if it was easy to customize the parameters of the spiking neural network, for example via a separate GUI.

  9. Like the first reviewer, I am not able try this out in a real application due to not having access to the kind of hardware that was used by the authors. Ideally, a third reviewer would assess performance in a similar context as the original one.

Summary

The manuscript and software must be both substantially expanded, and another review is required after the necessary changes have been made. Currently, the work appears to be more of a minor side product of other scientific work rather than a complete standalone software. This said, the software does have good potential to become a useful tool for the BCI community. While I am currently not able to recommend publication, I do strongly encourage the authors to continue development.

puolival commented 2 years ago

@oliviaguest It seems that I am not able to edit the checklist even though I am logged in.

eferrea commented 2 years ago

Hi @puolival (and hi @oliviaguest), many thanks for your review. It seems I can fix all of the reported issues.

Please bear in mind that the software was a necessary requirement (and NOT a minor side product) to perform BCI studies in our non-human primate labs and other different studies are now arising from this software. In addition, to the best of our knowledge, there is no software grouping together all the most important and necessary features for conducting intracortical BCI experiments in non-human primates, and despite some labs might have developed their own ad hoc solutions, scientists that want to start in this field might consider the software as a valuable option. Indeed please consider, that many important details of the methodological implementation are often not reported in scientific papers but need to be inferred from direct experience. Therefore, this software was developed and optimized under several years of direct experimental testing.

oliviaguest commented 2 years ago

@puolival I think you need to be reinvited by @whedon because you didn't accept the last time. No problem tho!

oliviaguest commented 2 years ago

@whedon add @puolival as reviewer

whedon commented 2 years ago

OK, @puolival is now a reviewer

danielskatz commented 2 years ago

@whedon remove @puolival as reviewer

We just need each reviewer once :)

whedon commented 2 years ago

OK, @puolival is no longer a reviewer

danielskatz commented 2 years ago

@whedon re-invite @puolival as reviewer

this is the way to get a new invitation...

whedon commented 2 years ago

The reviewer already has a pending invite.

@puolival please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

oliviaguest commented 2 years ago

Ooops, thanks @danielskatz!

danielskatz commented 2 years ago

If that link fails, tell me and I'll reissue it - there's a timing bug that sometimes happens here.

danielskatz commented 2 years ago

ok, @whedon didn't do what I wanted and removed both @puolival's - let's add one back

danielskatz commented 2 years ago

@whedon add @puolival as reviewer

whedon commented 2 years ago

OK, @puolival is now a reviewer

danielskatz commented 2 years ago

Whedon and the issue now are correct, so we just need to make sure @puolival is able to check off items by accepting the invitation, and if they cannot, we need to reissue it one more time, then it should work

oliviaguest commented 2 years ago

@danielskatz thank you!

eferrea commented 2 years ago

Thanks @puolival for your review and useful suggestions. We addressed at our best your concerns on the manuscript side as well as on the software side. We are confident that the suggested modifications helped to improve readability, and therefore should help increase software usage and usability. We also better clarified the purpose of the software and how it actually supports our research and how it can support the research of other investigators. In the following, you will find our answers to the specific questions and suggestions raised before. We hope that the revised document now fulfills the requirements and would be happy to learn that it is considered suitable for publication.

Manuscript

  1. The manuscript is currently too short. Please report results at least from one actual experiment. How good cursor control performance can be achieved using this software? Please also compare the performance to a typical state-of-the-art BCI experiment. Please also state more clearly the main functions of this software and how does it compare to the other existing software. Please also mention the most important limitations and/or how the software can be developed further. In its current version, after following the recommendations of the reviewer, the manuscript is already a little bit longer than 1000 words, which is the maximum allowed according to JOSS recommendations. We still added a short comment on control performance, but had to refrain from a systematic comparison with other BCI approaches, especially since such comparison would be of very limited value (see below).

The software supports the functioning of intracortical BCI experiments. It was developed and optimized under several years of direct experimental testing in two animals. We tried to make this information more clear in the revised version of the manuscript. Also, while it is evident that software for BCI experiments based on intracortical recordings must exist in several labs worldwide, we are not aware of any existing published software, especially presenting all the features that we implemented here. In the “statement of need”, we now emphasize even more that existing software makes use of EEG or ECog signals, which are very different in nature. Comparison of our software with cursor control performance from other approaches in principle is possible, but only based on published papers, since the software used for these experiments has not been made available publicly. Such comparison would not really be meaningful, though, since performance values strongly depend on several experimental parameters, for example, the quality of the recordings/brain implants, the number and task-selectivity of the neural signals extracted from the recordings, the behavioral training status of the subjects, etc.. As a rough quality measure, we now present values showing that in our recordings, the time to target acquisition was in the expected range when comparing hand movements and BCI control. We added now that one possible further development could be the implementation of nonlinear decoding methods (e.g LSTM, Transformers) to improve cursor control or the introduction of additional perturbation schemes for specific motor plasticity studies.

  1. It is unclear what is meant by the sentence “The code was successfully used from extracted neural activity of rhesus monkeys performing a similar center-out reach as previously described in a virtual-reality environment …”. If this means that experiments on rhesus monkeys were performed for the purpose of developing this software, the relevant ethical permissions, procedures, methods, etc. must be carefully reported according to standard good practices. It is not possible to recommend publication if an animal model was indeed used and these information are missing.

The software was only developed as a necessary step to perform translational studies in monkeys for future application in humans. All experiments where approved by the relevant authorities, and we added corresponding information on the permission to the manuscript. Since the BCI study was the primary goal of the physiological experiments, not the software development per se, we refrain from expanding the description of the physiological procedures here. Instead, the collected experimental data and BCI results will be published separately, including experimental procedures; this future work will then refer to the current software in the methods section.

  1. Related to points 1. and 2., is this software mainly intended to be used with recordings done on humans? This is currently unclear due to the reference to animal models. The abstract only refers to human patients. If the software is intended to be used with human patients, it must be tested with experiments involving human patients.

The software was not tested with human patients so far, but the framework is in principle suited for it. In Germany, to the best of our knowledge, the type of implants used for intracortical recordings are not approved yet for use in humans, therefore, we do not have access to respective patients and cannot test our software in humans. However, the software is used with Blackrock acquisition devices who also produce recordings systems used in human application. We added to the manuscript that the BCImat could be employed for other species. We revised the abstract to avoid the impression of human use.

  1. Does the statement “… movement stage information and position of the target are handled to clear the spike buffer regularly to avoid excessive memory overload” mean that the size of the buffer is dynamic during run-time? It would be much more preferable to use a constant-sized circular buffer (also known as circular queue or a ring buffer) instead, which would not have such issues.

The properties of the buffer are inherited from cbmex library for interfacing the Blackrock data acquisition system and cannot be chosen freely. We decided to maintain the same buffer structure when interfacing the artificial spiking neural network for maximal compatibility between application and simulation mode. We added corresponding information to the manuscript.

  1. Has this software been tested with any other acquisition system than Cereplex? The Cereplex system is now mentioned as an example but if the software is not (known to be) compatible with other acquisition systems, it should be clearly stated. The manuscript now gives an impression of a general-purpose system but if the software is in fact written for a specific hardware vendor, excluding the SNN part, this should be clearly communicated, perhaps already in the title.

The BCImat framework as such is general purpose, but it is true that one module, the spike buffer, is specific to and was tested with the Cereplex system only. As long as the same buffer structure is kept, which is also used in the simulation mode, the software can be used with other acquisition systems. We better clarify this in the text now.

  1. “We also implemented some important BCI features that are frequently used in the literature to provide efficient training and better decoding performance”. How much did the performance improve? How was the performance estimated? Please report the results that were obtained.

The idea of this software release is not to systematically study and quantitatively compare different neural decoding approaches/algorithms. Instead, it provides a software framework for users to implement such closed-loop experiments based on spiking data and to be able to simulate them without animal/patient. Due to the modularity, the user can then implement whichever decoder they prefer for their type of data and task. The statement about better decoding performance refers to published data (Gjlia et al. 2012) where it was found that online re-training of the decoder improves performance. Here we do not aim to innovate but make available some of the most important features that are used for this type of experiments. We rephrased the text to make it more clear.

  1. Please proofread the manuscript once more. In particular, please check that words that should be in singular form are not accidentally in plural form or vice versa. Spaces are sometimes missing before citation parentheses. Check that words are capitalized correctly (e.g. MATLAB is sometimes referred to as ‘Matlab’ and sometimes as ‘MATLAB’).

We now consistently use ‘Matlab’ and tried to catch all remaining typos/mistakes as carefully as possible.

  1. In the reference list, journal names are sometimes written in full and sometimes abbreviated. Please adhere to a consistent style. Thank you for pointing this out, we adjusted the references.

  2. Font sizes in Figure 1 are too small. Please ensure that all text is readable when the manuscript is printed.

Thank you, we used bigger fonts in the figure.

  1. Please write all abbreviations once in full.

We corrected any missing explanation of abbreviations.

  1. To make the manuscript more accessible, it would be good if it was explained what is a “standard c++ client-server application”.

Thank for this hint. Here, and throughout, we tried to avoid overly compact phrases and expanded on the text for easier readability.

  1. The main features of the SNNs must be described (which neuron model is used, size and connectivity of the network, is the network static or is there plasticity, etc.)

The network structure is simple, there is no connectivity among neurons. Each neuron fires with its own internal dynamic following the cosine model described in Georgopoulos, et al 1982. We added text for model clarification.

Software

  1. The license file contains the name of only one author while several authors have written the software (based on comments in the source files).

Ok, thanks, we added the full name list.

  1. There is no automated package management solution, which is a JOSS recommendation. Also, the installation instructions do not contain version numbers of external packages (e.g. VRPN), which is likely to cause issues sooner or later. The user also needs to manually copy files in order to get the software running, which is a step that I think should be automated.

We specified library versions that were tested for this project in the readme file. We modified the CMakelists.txt file in order to add the configuration file and DLLs automatically.

  1. There are no automated tests. The files “TestBlackrock.m” and “test_VRPN_connection.m” (neither of which is properly commented) can be used to test hardware/internet connections but there are no tests for assessing the functionality of this software. The authors should write at least unit tests for all functions.

A more detailed explanation of what to expect from these functions was now added to the readme and to the code comments. The readme was also updated providing information on how to test these functions.

  1. Many functions and methods are very long, might do several things, and are often not easy to understand. I suggest that long functions are refactored into several smaller ones, such that each function does one only thing (i.e. adherence to the so-called keep-it-simple principle).

The idea is that each function implements one task. We added more comments in cases where this strategy leads to extensive code.

  1. There are comments such as “… many of the parameters listed below are not used with the basic provided task controller but were used during our experiments”. It therefore appears that this software might not yet be suitable for general use, since it has not been written in a general-purpose context.

Even though the task controller is usually very specific to laboratory settings, we implemented here in TrackM a version of a standard center-out reach task used in most BCI studies. We left in the State_parser_class, a simple function that eventually can parse the messages from an eye-tracker.

  1. All functions, their input/output arguments, and non-trivial blocks of code should be properly commented. Indentation should be uniform across files. Single-letter variable names should not be used unless they are part of a mathematical equation.

We add more comments on non-trivial blocks of code and used proper identation. Also single letter variables whenever not being part of a formula or referring to figure objects were removed.

  1. Adhering to a specific style guide (see e.g. https://google.github.io/styleguide/) would make it easier to read the program code.

We tried to adhere to Matlab style Guidelines (https://www.mathworks.com/matlabcentral/fileexchange/46056-matlab-style-guidelines-2-0) . One exception here is that functions were named according to this style compute_diff() , rather than this style computeDiff(). Since in the previous version both styles were interleaved, we changed the code consistently.

  1. The software would be more valuable for researchers if it was easy to customize the parameters of the spiking neural network, for example via a separate GUI.

At the moment only the number of neurons can be decided as input. It seems not a big issue to enter this number inside the launching function. We consider a GUI potentially useful for the NN should we decide to add more parameters that can be configured (e.g., spiking noise, modulation depth).

  1. Like the first reviewer, I am not able try this out in a real application due to not having access to the kind of hardware that was used by the authors. Ideally, a third reviewer would assess performance in a similar context as the original one.

We include the simulation mode in this BCI framework exactly for the reason that users can test the software even before starting with physiological experiments.

puolival commented 2 years ago

@eferrea Thanks for the detailed response. Where can I access the updated manuscript (pdf)?

eferrea commented 2 years ago

@puolival, I now uploaded the paper.pdf in the Github "paper" folder.

mstimberg commented 2 years ago

To get the rendered PDF here we should also be able to use

@whedon generate pdf

mstimberg commented 2 years ago

@whedon generate pdf

[sorry, keep forgetting that whedon commands need to start a comment]

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

eferrea commented 2 years ago

Hi @puolival and (hi @oliviaguest), thanks again for reviewing the manuscript. Since we have a manuscript and a grant pending on this paper I would ask if a much longer time is needed for the review so that in case we will go ahead with the other submissions. Please let me also know if meanwhile, I could provide additional information to agevolate the process. Thanks again for your work and best regards.

mstimberg commented 2 years ago

Hi everyone. In case there are any doubts about this: I still recommend the acceptance of the paper in its most recent version. Quite obviously @puolival's review was more "demanding" than my own, but I also agree that the changes to address the raised points improved the manuscript/code/documentation.

I hope I am not overstepping my role as a reviewer here, but let me mention two points from the JOSS guidelines that might affect the evaluation of the current manuscript and code:

Your paper must not focus on new research results accomplished with the software.

(Submission guidelines for authors) I think it is important to keep in mind that the role of papers in JOSS is different from a "software paper" in most other journals.

Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.

Good: An automated test suite hooked up to continuous integration (GitHub Actions, Circle CI, or similar) OK: Documented manual steps that can be followed to objectively check the expected functionality of the software (e.g., a sample input file to assert behavior)

(Review criteria: tests) While I of course agree that unit tests are important, I also acknowledge that the core part of the presented work involves GUI and client-server communication, which means that a fully automated test suite presents a considerable effort.