[REVIEW]: Frites: A Python package for functional connectivity analysis and group-level statistics of neurophysiological data

[whedon]

Submitting author: !--author-handle-->@EtienneCmb<!--end-author-handle-- (Etienne Combrisson) Repository: https://github.com/brainets/frites Branch with paper.md (empty if default branch): paper Version: v0.4.3 Editor: !--editor-->@meg-simula<!--end-editor-- Reviewers: @madvn, @travisbthomp Archive: 10.5281/zenodo.7278278

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/437a7362501b2ea984e1d4fed4646076"><img src="https://joss.theoj.org/papers/437a7362501b2ea984e1d4fed4646076/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/437a7362501b2ea984e1d4fed4646076/status.svg)](https://joss.theoj.org/papers/437a7362501b2ea984e1d4fed4646076)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@madvn & @travisbthomp, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @meg-simula know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @madvn

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@EtienneCmb) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] *Summary:** Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @travisbthomp

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@EtienneCmb) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @madvn, @travisbthomp it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

PDF failed to compile for issue #3842 with the following error:

 Can't find any papers to compile :-(

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.16 s (1281.4 files/s, 115986.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                         109           2415           5772           7325
SVG                              1              0              0            710
reStructuredText                85            635            726            590
TeX                              1             25              0            221
YAML                             6             45              5            181
CSS                              1             15              3             49
HTML                             2              2              3             28
DOS Batch                        1              8              1             26
make                             2              8             11             23
-------------------------------------------------------------------------------
SUM:                           208           3153           6521           9153
-------------------------------------------------------------------------------

Statistical information for the repository 'cf89deaa1b9fbcf1596c55f5' was
gathered on 2021/10/22.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
EtienneCmb                     387         24295           8358           99.59
Ruggero Basanisi                 2             9              8            0.05
brovelli.a                      11            71             45            0.35

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
EtienneCmb                15464           63.7         11.0               19.01
Ruggero Basanisi              8           88.9          9.5               50.00
brovelli.a                   40           56.3          7.1               35.00

[meg-simula]

👋🏼 @EtienneCmb @madvn @travisbthomp this is the review thread for the paper. All of our communications will happen here from now on.

Both reviewers have checklists at the top of this thread with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. Also note that the current paper is in the branch 'paper', so use e.g.

-> @whedon generate pdf from branch paper -> @whedon check references from branch paper

Please feel free to ping me (@meg-simula ) if you have any questions/concerns.

[meg-simula]

@whedon generate pdf from branch paper

[whedon]

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

[meg-simula]

@whedon check references from branch paper

[whedon]

Attempting to check references... from custom branch paper

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1002/hbm.23471 is OK
- 10.3389/fnins.2013.00267 is OK
- 10.1145/2833157.2833162 is OK
- 10.1371/journal.pcbi.1008302 is OK
- 10.21105/joss.01081 is OK

MISSING DOIs

- 10.5334/jors.148 may be a valid DOI for title: xarray: ND labeled arrays and datasets in Python
- 10.1523/jneurosci.4892-14.2015 may be a valid DOI for title: Characterization of Cortical Networks and Corticocortical Functional Connectivity Mediating Arbitrary Visuomotor Mapping
- 10.1101/2021.08.14.456339 may be a valid DOI for title: Group-level inference of information-based measures for the analyses of cognitive brain networks from neurophysiological data

INVALID DOIs

- None

[brovelli]

We checked the 3 MISSING DOIs on https://dx.doi.org/ and they all appear to be correct.

[whedon]

:wave: @travisbthomp, please update us on how your review is going (this is an automated reminder).

[whedon]

:wave: @madvn, please update us on how your review is going (this is an automated reminder).

[madvn]

Sorry about the delay, I will get to it this weekend.

[travisbthomp]

Getting around to this now with other deadlines mostly behind me.

[travisbthomp]

@EtienneCmb I am running some of the examples from the documentation. I noticed the following (likely unintentional) output from plot_stim_spec_network.py

plot_stim_spec_network.py:211: SyntaxWarning: "is" with a literal. Did you mean "=="? if direction is 'x->y': plot_stim_spec_network.py:213: SyntaxWarning: "is" with a literal. Did you mean "=="? elif direction is 'y->x': plot_stim_spec_network.py:215: SyntaxWarning: "is" with a literal. Did you mean "=="? elif direction is 'x.y': plot_stim_spec_network.py:260: SyntaxWarning: "is" with a literal. Did you mean "=="? if direction is 'x->y': plot_stim_spec_network.py:262: SyntaxWarning: "is" with a literal. Did you mean "=="? elif direction is 'y->x': plot_stim_spec_network.py:264: SyntaxWarning: "is" with a literal. Did you mean "=="? elif direction is 'x.y':

[travisbthomp]

@EtienneCmb

All Issues in this comment have now been addressed. Nov 29, 2021

Resolution: Updated to frites 0.4.1 and re-checked the scripts below

1. The second figure generated by plot_wf_mi_cc.py does not match the second figure in the corresponding documentation. (see https://brainets.github.io/frites/auto_examples/mi/plot_wf_mi_cc.html)

2. The example plot_est_custom.py does not work for me and generates the following error (see https://brainets.github.io/frites/auto_examples/estimators/plot_est_custom.html)

Traceback (most recent call last): File "plot_est_custom.py", line 12, in from frites.estimator import CustomEstimator ImportError: cannot import name 'CustomEstimator' from 'frites.estimator'

3. The example plot_est_comparison.py does not faithfully recreate figure two (see https://brainets.github.io/frites/auto_examples/estimators/plot_est_comparison.html)

4. The example plot_ccf.py does not work for me and generates the following error (see https://brainets.github.io/frites/auto_examples/conn/plot_ccf.html)

Traceback (most recent call last): File "plot_ccf.py", line 11, in from frites.conn import conn_ccf ImportError: cannot import name 'conn_ccf' from 'frites.conn'

5. The example plot_ar_instantaneous.py: The 3rd figure in the documentation (see https://brainets.github.io/frites/auto_examples/armodel/plot_ar_instantaneous.html) should be updated to match the figure generated by the code (different subplot titles, etc)

Update note: removed some other comments regarding plot mismatches as I noticed the data was generated using random values.

[madvn]

Hello all, I am just getting to this as well. Sorry about the delay.

Looks like a very interesting toolbox. I think the idea of feature-specific functional networks using mutual information can be used widely in not just neuroscientific settings but also sociology or even, more topically, epidemiology. I've updated my checklist based on a quick look at the repo and installing the package. I also took a quick look at the paper and have some notes for the authors based on the following requirements while I work my way through the rest of the checklist.

- [ ] **Summary:** Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

You have a summary but please revise it to address the "for a diverse, non-specialist audience" part of the requirement. Add a note about the breadth of usability of these tools in other domains besides neuroscience.

- [ ] **State of the field:** Do the authors describe how this software compares to other commonly-used packages?

Please place your tool in the space of other information theory tools such as infotheory, IDTXL etc. as well as network neuroscience tools such as Brain connectivity toolbox etc. I do believe there are unique aspects to your package, such as combining information-theoretic measures with network neuroscience to find feature-specific functional networks (I'm not aware of other tools that does this -- I will look it up. If you are aware of others, please include those.

- [ ] **References:** Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper [citation syntax]( https://rmarkdown.rstudio.com/authoring_bibliographies_and_citations.html#citation_syntax)?

Please update references based on additional paper that you will cite to address the above points.

[EtienneCmb]

Dear @travisbthomp,

plot_stim_spec_network.py:211: SyntaxWarning: "is" with a literal. Did you mean "=="?

I fixed this morning the warning message by replacing 'is' with '=='

The second figure generated by plot_wf_mi_cc.py does not match the second figure in the corresponding documentation.

Indeed, as you saw, we are mainly using random data for illustrating the functions

The example plot_est_custom.py does not work for me and generates the following error The example plot_ccf.py does not work for me and generates the following error

Indeed, both examples were using the develop branch. This morning I pushed the version 0.4.1 of Frites to Pypi, it should works now once updated.

[travisbthomp]

@whedon generate pdf from branch paper

[whedon]

Attempting PDF compilation from custom branch paper. Reticulating splines etc...

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[travisbthomp]

Suggestions for paper

After reviewing the paper I would like to make a few comments in the spirit of the review guidelines above.

Summary

Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

The description of the high level functionality is not quite clear. "Frites provides such an integrated framework" for what? Is Frites limited to "discovery of cognitive brain networks from multi-channel neurophysiological datasets"? What types of networks? Conversely, I think that the documentation text is quite approachable and nice!

Frites is a Python toolbox for assessing information-based measures on human and animal neurophysiological data (M/EEG, Intracranial). The toolbox also includes directed and undirected connectivity metrics such as group-level statistics on measures of information (information-theory, machine-learning and measures of distance).

In short, the current summary seems obfuscated and potentially difficult to read for a "non-specialist audience".

Statement of need

Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?

The paper does indeed have a section titled 'Statement of Need'. However, this section does not establish a clear need for the software nor does it "clearly state what problems the software is designed to solve and who the target audience is". At least, it does not do these things in a way that can be appreciated by an non-expert in the area, such as myself, who is not currently aware of the state of the art approaches, what problems they could solve better, and why Frites fills any of these gaps.

I would suggest revisiting this write up with the journal guidelines in mind.

State of the field

Do the authors describe how this software compares to other commonly-used packages?

I cannot find where this has been done in a manner that is decipherable to a non-expert. I would suggest an approach which clearly states the central problem Frites aims to solve and endeavors a comparative discussion to other available tools; what are the advantages of Frites over other packages? If there are no other packages, simply state that Frites is the first of its kind.

Quality of writing

Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?

The authors do not struggle with the tenets of structural composition. The central message of the statement of need is not currently clear, but this is not an issue with the sentence-by-sentence structure/language/quality per se.

References

Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

I did not see any issues with the references and all provided DOI links correct redirected to the appropriate paper.

[EtienneCmb]

Thank you @madvn and @travisbthomp for the comments on the manuscript, I do agree with both of you about the lack of accessibility. I'll work on the manuscript soon to clarify the text and make it as accessible as possible to the non-expert.

[travisbthomp]

Functionality Review

Functionality

Have the functional claims of the software been confirmed?

It is not currently clear what primary claims should be assessed. I was able to piece together a possibly-incomplete list from the documentation and accompanying paper.

From the documentation I can see a few things:

1. Frites is a Python toolbox for assessing information-based measures on human and animal neurophysiological data (M/EEG, Intracranial)
2. Extract task-related cognitive brain networks : i.e brain regions and connectivity between brain regions that are modulated according to the task
3. Group-level statistical inferences : being able to draw conclusions on a population of subjects while taking into account the the richness of data at the single subject level
4. With Frites, we try to isolate the brain activity that is either related to the stimulus or to the behavior.

From the paper, there are a few more claims to be assessed:

1. this package can be used to study the relation between local brain activity, cross-frequency coupling and inter-areal functional connectivity (FC) with experimental variables
2. For what concerns FC, the toolbox allows the estimate of a. dynamic (i.e., time-resolve) functional connectivity b. undirected (e.g., mutual information) functional connectivity and c. directed (e.g. Granger causality) functional connectivity on a single-trial basis
3. The package integrates a non-parametric permutation-based statistical framework to perform group-level inferences on non negative measures of information.
4. The toolbox includes different methods that cope with multiple-comparison correction problems such as test- and cluster-wise p-value corrections.
5. The implemented framework supports both fixed- and random-effect models to adapt to inter-individuals and inter-sessions variability

The authors should review this list to make sure that I have not misrepresented, or missed any of, their functionality claims. It seems that verifying these functionality claims would be easiest by pointing to the corresponding example code. To make these claims easier to verify I would suggest considering a strategy similar to the following:

On the main documentation page it might be natural to have a section called something like "What Frites can do for you" between "Description" and "Highlights" which lists the functionality claims and direct link to a corresponding script from the examples section. For instance

"What Frites can do for you"

1. Frites can assess information-based measures on human and animal neurophysiological data (as a link to an example script)
2. Frites can extract task-related cognitive brain networks (as a link to an example script)
3. Frites can conduct a group-level statistical inference on time-series data (as a link to an example script)
4. Frites can be used to study relationships between local brain activity and cross-frequency coupling (link to example script)
5. Frites can also highlight links between inter-areal functional connectivity with experimental variables (link to example script) and so forth.

Essentially, each claim can be made explicit and each claim can be linked to a demonstration of that claim vis-a-vis one of the example scripts.

Performance

If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

There are a few performance-related mentions that I can find in the paper. They are as follows (performance-related words are in bold text)

Frites is optimally designed for the analysis of continuous and multi-channel neurophysiological data, encompassing recordings with either uniform spatial sampling (e.g., M/EEG data) and spatially sparse intracranial recordings, such as intracranial EEG or Local Field Potentials (LFPs).

The claim here appears to be that the design of Frites is optimal when considering the task of analyzing particular types of data sets. What does optimal mean here and where can evidence be found for this claim? I did not see, for instance, a documentation example where Frites is compared to other software to show that it is "optimal" amongst the current choices.

Maybe this claim could be made more explicit and confirmed as part of the documentation via a "Performance" page that could be linked off the main bar at the top of the documentation (which currently holds "Overview" "Installation" "API Reference" "Examples")

In order to facilitate automated and efficient usage, Frites provides a set of high-level workflows that integrate several analysis steps from information-based estimation to network level statistical inference.

Similar to above. Where do I find any real information about what workflows really are, let alone how they are automated and enable more efficiency? I assume that would be here but this page only has a cursory discussion of workflows and leaves me with no real concept of what they actually are beyond whatever my intuition suggests from the name "workflow". There is an "in a nutshell" discussion which is not particuarly helpful in addition to a link to "chekout those examples" which takes me to the main examples page so that I must dig to find examples that demonstrate the workflows themselves. Moreover, the links to frites.dataset, frites.workflow and frites.conn on the overview page (above the workflow section) all end in a 404.

The burden of figuring out what a workflow is, and substantiating the claims of automation and efficiency should not be on the reader.

Since several computations implemented in the workflows, such as permutation tests, are computationally demanding, Frites natively supports parallel processing using the Joblib package.

Which computations are the most demanding? Can we point to the examples directly in addition to any performance gains from using Joblib? This would fit well on a "Performance" documentation page as mentioned above

In addition, some functions can further be accelerated using the Numba compiler as an optional dependency. Programming optimizations and external dependencies allow to investigate large-scale datasets in a reasonable time.

What functions can be accelerated / where is acceleration most profitable in Frites? To justify the claim of "large scale datasets in reasonable time" it seems natural to include a benchmark plot showing the processing time of Frites, versus other software packages as well, against a data set size.

[meg-simula]

@EtienneCmb @travisbthomp @madvn Thanks for your reviews and comments thus far. Could you please update us on the status of your review/responses?

[travisbthomp]

Dear @meg-simula, thank you for updating this thread. I am personally waiting on a response from @EtienneCmb (c.f. my previous remarks) before proceeding further. If I should continue to take some action now please don't hesitate to let me know.

[meg-simula]

@EtienneCmb We seem to be awaiting your revisions before proceeding with the submission. Could you please give us an update?

[meg-simula]

@EtienneCmb We still seem to be awaiting your revisions before proceeding with the submission. Could you please give us an update?

[brovelli]

Dear Marie,

Sorry for the delay, we are about to submit the revised version and reply to the comments of the reviewers.

Best regards

[EtienneCmb]

Dear @meg-simula,

I'm sincerely sorry for the very late answer, I was absorbed in other projects. I'll past out answers to the reviewers today or tomorrow.

Best,

[EtienneCmb]

@whedon generate pdf from branch paper

[editorialbot]

My name is now @editorialbot

[EtienneCmb]

Dear @meg-simula and @travisbthomp, please find below our detailed answers to the comments about the paper and about functionality.

Suggestions for paper

Summary

In short, the current summary seems obfuscated and potentially difficult to read for a "non-specialist audience".

We agree with the reviewer that the first version of the summary was not accessible to a non-specialist audience. We simplified the section to be closer to the online documentation. In addition, the Summary section now only includes the context and the problematic that Frites tries to solve.

Statement of need

The paper does indeed have a section titled 'Statement of Need'. However, this section does not establish a clear need for the software nor does it "clearly state what problems the software is designed to solve and who the target audience is". At least, it does not do these things in a way that can be appreciated by an non-expert in the area, such as myself, who is not currently aware of the state of the art approaches, what problems they could solve better, and why Frites fills any of these gaps.

We improved the Statement of need section with the following points :

1. We now clearly state that Frites has been designed for researchers and students with little programming knowledge and working on the discovery of cognitive brain networks using neurophysiological data
2. We explain that the software comes with measures of information to find complex relations between brain data and external variables and also that those measures of information are combined with statistics that adapt to the variability across participants

All of those points are then further detailed in the individual sections.

State of the field

I cannot find where this has been done in a manner that is decipherable to a non-expert. I would suggest an approach which clearly states the central problem Frites aims to solve and endeavors a comparative discussion to other available tools; what are the advantages of Frites over other packages? If there are no other packages, simply state that Frites is the first of its kind.

Indeed, it was a missing information in the previous manuscript. We now added the sentence : To the best of our knowledge, Frites is the only Python software combining information-theoretical approaches with flexible group-level analyses for the discovery of cognitive brain networks.

Quality of writing

The authors do not struggle with the tenets of structural composition. The central message of the statement of need is not currently clear, but this is not an issue with the sentence-by-sentence structure/language/quality per se.

We hope that this new version clarifies the Statement of need.

Functionality Review

Functionality

On the main documentation page it might be natural to have a section called something like "What Frites can do for you" between "Description" and "Highlights" which lists the functionality claims and direct link to a corresponding script from the examples section.

We thank the reviewer for the in-depth analysis of the functionality of Frites and proposal for amelioration. As suggested, we added a section in the main documentation page entitled “What can you do with Frites?” between "Description" and "Highlights" which lists the functionality claims and direct link to a corresponding overview and script from the examples section.

Performance

The claim here appears to be that the design of Frites is optimal when considering the task of analyzing particular types of data sets. What does optimal mean here and where can evidence be found for this claim? I did not see, for instance, a documentation example where Frites is compared to other software to show that it is "optimal" amongst the current choices.

We removed the term “optimal” from the paragraph. Our intention was not to make any claim about optimality, but rather to suitability. The new paragraph now reads as follows:

Frites is suitable for the analysis of continuous and multi-channel neurophysiological data, encompassing recordings with either uniform spatial sampling (e.g., M/EEG data) and spatially sparse intracranial recordings, such as intracranial EEG or Local Field Potentials (LFPs).

Similar to above. Where do I find any real information about what workflows really are, let alone how they are automated and enable more efficiency? I assume that would be here but this page only has a cursory discussion of workflows and leaves me with no real concept of what they actually are beyond whatever my intuition suggests from the name "workflow". There is an "in a nutshell" discussion which is not particularly helpful in addition to a link to "checkout those examples" which takes me to the main examples page so that I must dig to find examples that demonstrate the workflows themselves.

We do agree with the reviewer that the description of the WorkFlows was not sufficient. We now added a clearer description of the workflows in the section Step by step presentation of the workflow.

In addition, we removed any claim of efficient usage from the text to avoid any mis-understanding. In fact, we meant that workflows are higher-level scripts that can be used to run multiple analysis steps.

Moreover, the links to frites.dataset, frites.workflow and frites.conn on the overview page (above the workflow section) all end in a 404.

The dead links on the overview page should now all be working as expected.

Which computations are the most demanding? Can we point to the examples directly in addition to any performance gains from using Joblib? This would fit well on a "Performance" documentation page as mentioned above

In addition, some functions can further be accelerated using the Numba compiler as an optional dependency. Programming optimizations and external dependencies allow to investigate large-scale datasets in a reasonable time.

What functions can be accelerated / where is acceleration most profitable in Frites? To justify the claim of "large scale datasets in reasonable time" it seems natural to include a benchmark plot showing the processing time of Frites, versus other software packages as well, against a data set size.

We thank the reviewer for this great idea of illustrating the performance of Frites. We therefore included a Performance page on the website, in the overview section. This section explains that we are mainly using three tricks to decrease computing time :

• Tensor computations illustrated here
• Parallel computing. This one is not trivial to show live in the documentation, mainly because gains appear with larger data and even more on computer clusters, which is not feasible on the documentation. Also, an example running on a laptop could potentially led to very different results depending on the OS and materials.
• Code compilation : we specifically pointed toward the functions that can be compiled if Numba is installed.

We want to thank again @travisbthomp for this detailed review. We hope that those revisions are going to clarify the need for this software, even for non-expert.

[EtienneCmb]

@editorialbot generate pdf from branch paper

[editorialbot]

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@editorialbot commands

[EtienneCmb]

@editorialbot set paper as branch

[editorialbot]

Done! branch is now paper

[EtienneCmb]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[EtienneCmb]

Dear @madvn,

Looks like a very interesting toolbox. I think the idea of feature-specific functional networks using mutual information can be used widely in not just neuroscientific settings but also sociology or even, more topically, epidemiology. I've updated my checklist based on a quick look at the repo and installing the package. I also took a quick look at the paper and have some notes for the authors based on the following requirements while I work my way through the rest of the checklist.

We thank the reviewer for this great remark and for the positive comments. We will try to address the concerns raised.

You have a summary but please revise it to address the "for a diverse, non-specialist audience" part of the requirement. Add a note about the breadth of usability of these tools in other domains besides neuroscience.

We agree that the first version of the “Summary” section was hard to grasp for non-expert readers. We simplified this section and also clarified why measures of information are important for brain data analyses. About the breadth of usability, we added the following sentence to the Measures of information and group-level statistics section :

While Frites is strongly oriented toward neuroscience applications, the estimators of information are agnostic to data types and can therefore be applied to different fields.

Please place your tool in the space of other information theory tools such as infotheory, IDTXL etc. as well as network neuroscience tools such as Brain connectivity toolbox etc. I do believe there are unique aspects to your package, such as combining information-theoretic measures with network neuroscience to find feature-specific functional networks (I'm not aware of other tools that does this -- I will look it up. If you are aware of others, please include those.

We thank the reviewer for pointing us toward infotheory and the possible link between Frites and the BCT. We added the following sentence to the revised manuscript for encompassing those packages :

The networks built from the FC can then be further analyzed using graph-theoretical approaches from the Brain Connectivity Toolbox (Rubinov and Sporns, 2010). Nevertheless, Frites is not limited to the included estimators as the definition of custom ones is also supported, such as IDTxl kernel methods (Wollstadt et al, 2019), infotheory information decomposition (Candadai & Izquierdo, 2019) or scikit-learn cross-validated classifiers (Pedregosa et al., 2011). The estimated information can then be combined with the group-level statistics for the identification of feature-specific networks. While Frites is strongly oriented toward neuroscience applications, the estimators of information are agnostic to data types and can therefore be applied to different fields.

We sincerely thank the reviewer for the feedback,

[EtienneCmb]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[brovelli]

Dear @meg-simula,

Is there anything else we need to do to advance the reviewing process?

Thanks.

Best regards

[travisbthomp]

Dear @EtienneCmb @brovelli (CC: @meg-simula),

Thank you for making changes pursuant to the review of Frites. Regarding advancing the reviewing process (@brovelli): I am currently traveling overseas; I will be happy to review the comments left by @EtienneCmb when I return in a few weeks.

[madvn]

Thank you for your responses @EtienneCmb .. I will get back to you, in the next week.

[meg-simula]

@EtienneCmb @madvn @travisbthomp Thanks for the updates and responses! Hopefully everyone is back from summer now, and we can get this review sorted out. @madvn and @travisbthomp Could you please have another look at the revised manuscript and software, and concretely, your reviewer checklists above?

[travisbthomp]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[travisbthomp]

@EtienneCmb @brovelli @meg-simula

I have completed my review of Frites for this round. I read through the changes and I wanted to thank both @EtienneCmb @brovelli for their work addressing readability issues in addition to severe documentation shortfalls (such as dead links and examples that did not run, etc). I also noted that the authors re-evaluated their performance claims and added a "Performance" documentation page, which is a nice addition since it draws the reader's attention to Joblib and Numba. The reviewers also clarified the current state of the art and Frites' contextual position within this area.

Due to the author's efforts, I have now checked 14 / 19 of the review checklist items. Below, I have detailed each item in the checklist that was checked this round or left unchecked. Each item is marked as Resolved (checked) or includes a To Resolve suggestion. One item is marked Resolved (with a suggestion) indicating that it has been checked but that the authors might consider / reflect upon the suggestion at their discretion.

Review Checklist Items

• General Checks

  1. Does this submission meet the scope eligibility described in the JOSS guidelines

  Resolved I believe that Frites meets the bar for a substantial scholarly effort.

• Functionality

  1. Have the functional claims of the software been confirmed?

  Partially. The authors have demonstrated many of the capabilities of the software in their examples section. However, I have raised an issue with one of the two explicitly stated goals of the software. As the functional claim verification occurs in the documentation as part of the examples, I defer this comment to item (5) of the next section

  To resolve: See item (5) of the next section.

  2. If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

  Resolved The authors addressed this in their review response. They have removed the performance claims that were brought up in the first round of review.

• Documentation

  1. Do the authors clearly state what problems the software is designed to solve and who the target audience is?

  The Overview page of the documentation clearly states two goals for Frites. No target audience is stated in the documentation.

  To Resolve: Please add a target audience statement to the documentation.

  2. Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.

  Resolved Yes. The authors provide an installation main page that lists all dependencies and includes instructions for installation with Pip.

  3. Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).

  Resolved Many examples are included. These are mostly synthetic, but it is understandable that the authors may not have access / permission to release raw data sets. The authors might consider illustrating Frites on some freely available EEG data or similar.

  4. Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

  Resolved (with a suggestion) I have marked this as resolved as the authors have included enough examples (in the examples section) to meet the very barest of minimal requirements as the examples do contain the API objects if one digs through them. I do have a suggestion, though. In the "API" section of the documentation you list several API objects/items (Dataset, Workflows, Connectivity, etc). Clicking on these items (e.g. the frites.workflow link) takes me to a separate page. It would be useful to, at the very least, include links to the examples that use each cited API (e.g. WfMi, WfMiCombine, WfConnComod, etc) object on this landing page. In this way, the users can at least see a demonstration of these various API and get some minimal context for how they are used. The API documentation pages are currently as sparse as possible while still being minimally feasible for publication. If the authors want others to actually use their software, post-publication, the API documentation could be made more verbose. This is especially true given the author's explicit claim to be targeting "researchers and students with little programming knowledge".

  5. Are there automated tests or manual steps described so that the functionality of the software can be verified?

  The Frites documentation currently ships with several examples. As far as I can see, these currently serve as the "automated tests" that allow for verification of the software functionality. The "Goals" page lists two goals of Frites: (A) Extract task-related cognitive brain networks and (B) Group-level statistical inferences. As a reviewer, I expect the main goals for the functionality to have verifiable examples / resources available. One can argue that (B) is verifiable as there are seven examples of statistical tests in the "Examples" section (for two variables only). However, I cannot find anywhere where a task-related cognitive brain network is extracted / produced. I do see that the authors have included "Autoregressive Models" examples. I looked through these and they do show how Granger causality can be used to infer connectivity but they use synthetic networks a priori. From the documentation goals, one would expect a set of a-priori signal data to be used to create a graph from the GC analysis. I suspect that this would be one of the main tasks that a user of Frites would want to carry out.

  To Resolve: To resolve this issue, a clear verification of the author's stated goal (A) seems to be needed. One thing that I can think of would be to create and release a data set (using, e.g. StimSpecAR) of node-wise signals, reflecting say an ~8-10 node graph, and create and example that imports the nodal stimulus data set and then re-creates the connectivity and outputs the weighted graph (or correlation matrix or both) from those signals. The authors may have a better idea.

  6. Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

  I do not see any anything explicit in this direction on the online documentation.

  To resolve: Add an explicit mention of this item in line with the review checklist requirements.

• Software Paper

  1. Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

  The authors revised and simplified the summary section for readability. I think this section is readable by a non-specialist audience. However, this section lacks "a clear description of the high-level functionality and purpose of the software". One could argue that such a description of purpose is provided: (a) implicitly by the final sentence about the "[need for] powerful measures of information to detect complex relations and statistical relevance" (but this is vague) or (b) in the statement of need (but this is not in the summary section, as required by the guidelines)

  To resolve: please include "a clear description of the high-level functionality and purpose of the software" in the summary section as requested by the checklist.

  2. Statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?

  Resolved The statement of need now includes a target audience statement in addition to stating that [Frites is for those] working on the discovery of cognitive brain networks using neurophysiological data. The high-level functionality discussed are: information based analysis, group-level statistical inference, functional connectivity measurement. In light of these observations, I would again cite the need for the authors to demonstrate (e.g. with an example in the documentation) how to "construct a cognitive brain network using neurophysiological data".