Closed whedon closed 1 year ago
Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @madvn, @travisbthomp it looks like you're currently assigned to review this paper :tada:.
:warning: JOSS reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
:star: Important :star:
If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿
To fix this do the following two things:
For a list of things I can do to help you, just type:
@whedon commands
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
@whedon generate pdf
PDF failed to compile for issue #3842 with the following error:
Can't find any papers to compile :-(
Software report (experimental):
github.com/AlDanial/cloc v 1.88 T=0.16 s (1281.4 files/s, 115986.2 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Python 109 2415 5772 7325
SVG 1 0 0 710
reStructuredText 85 635 726 590
TeX 1 25 0 221
YAML 6 45 5 181
CSS 1 15 3 49
HTML 2 2 3 28
DOS Batch 1 8 1 26
make 2 8 11 23
-------------------------------------------------------------------------------
SUM: 208 3153 6521 9153
-------------------------------------------------------------------------------
Statistical information for the repository 'cf89deaa1b9fbcf1596c55f5' was
gathered on 2021/10/22.
The following historical commit information, by author, was found:
Author Commits Insertions Deletions % of changes
EtienneCmb 387 24295 8358 99.59
Ruggero Basanisi 2 9 8 0.05
brovelli.a 11 71 45 0.35
Below are the number of rows from each author that have survived and are still
intact in the current revision:
Author Rows Stability Age % in comments
EtienneCmb 15464 63.7 11.0 19.01
Ruggero Basanisi 8 88.9 9.5 50.00
brovelli.a 40 56.3 7.1 35.00
👋🏼 @EtienneCmb @madvn @travisbthomp this is the review thread for the paper. All of our communications will happen here from now on.
Both reviewers have checklists at the top of this thread with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.
The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.
We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. Also note that the current paper is in the branch 'paper', so use e.g.
-> @whedon generate pdf from branch paper -> @whedon check references from branch paper
Please feel free to ping me (@meg-simula ) if you have any questions/concerns.
@whedon generate pdf from branch paper
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
@whedon check references from branch paper
Attempting to check references... from custom branch paper
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1002/hbm.23471 is OK
- 10.3389/fnins.2013.00267 is OK
- 10.1145/2833157.2833162 is OK
- 10.1371/journal.pcbi.1008302 is OK
- 10.21105/joss.01081 is OK
MISSING DOIs
- 10.5334/jors.148 may be a valid DOI for title: xarray: ND labeled arrays and datasets in Python
- 10.1523/jneurosci.4892-14.2015 may be a valid DOI for title: Characterization of Cortical Networks and Corticocortical Functional Connectivity Mediating Arbitrary Visuomotor Mapping
- 10.1101/2021.08.14.456339 may be a valid DOI for title: Group-level inference of information-based measures for the analyses of cognitive brain networks from neurophysiological data
INVALID DOIs
- None
We checked the 3 MISSING DOIs on https://dx.doi.org/ and they all appear to be correct.
:wave: @travisbthomp, please update us on how your review is going (this is an automated reminder).
:wave: @madvn, please update us on how your review is going (this is an automated reminder).
Sorry about the delay, I will get to it this weekend.
Getting around to this now with other deadlines mostly behind me.
@EtienneCmb I am running some of the examples from the documentation. I noticed the following (likely unintentional) output from plot_stim_spec_network.py
plot_stim_spec_network.py:211: SyntaxWarning: "is" with a literal. Did you mean "=="? if direction is 'x->y': plot_stim_spec_network.py:213: SyntaxWarning: "is" with a literal. Did you mean "=="? elif direction is 'y->x': plot_stim_spec_network.py:215: SyntaxWarning: "is" with a literal. Did you mean "=="? elif direction is 'x.y': plot_stim_spec_network.py:260: SyntaxWarning: "is" with a literal. Did you mean "=="? if direction is 'x->y': plot_stim_spec_network.py:262: SyntaxWarning: "is" with a literal. Did you mean "=="? elif direction is 'y->x': plot_stim_spec_network.py:264: SyntaxWarning: "is" with a literal. Did you mean "=="? elif direction is 'x.y':
@EtienneCmb
The second figure generated by plot_wf_mi_cc.py does not match the second figure in the corresponding documentation. (see https://brainets.github.io/frites/auto_examples/mi/plot_wf_mi_cc.html)
The example plot_est_custom.py does not work for me and generates the following error (see https://brainets.github.io/frites/auto_examples/estimators/plot_est_custom.html)
Traceback (most recent call last):
File "plot_est_custom.py", line 12, in
The example plot_est_comparison.py does not faithfully recreate figure two (see https://brainets.github.io/frites/auto_examples/estimators/plot_est_comparison.html)
The example plot_ccf.py does not work for me and generates the following error (see https://brainets.github.io/frites/auto_examples/conn/plot_ccf.html)
Traceback (most recent call last):
File "plot_ccf.py", line 11, in
Update note: removed some other comments regarding plot mismatches as I noticed the data was generated using random values.
Hello all, I am just getting to this as well. Sorry about the delay.
Looks like a very interesting toolbox. I think the idea of feature-specific functional networks using mutual information can be used widely in not just neuroscientific settings but also sociology or even, more topically, epidemiology. I've updated my checklist based on a quick look at the repo and installing the package. I also took a quick look at the paper and have some notes for the authors based on the following requirements while I work my way through the rest of the checklist.
- [ ] **Summary:** Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
You have a summary but please revise it to address the "for a diverse, non-specialist audience" part of the requirement. Add a note about the breadth of usability of these tools in other domains besides neuroscience.
- [ ] **State of the field:** Do the authors describe how this software compares to other commonly-used packages?
Please place your tool in the space of other information theory tools such as infotheory
, IDTXL
etc. as well as network neuroscience tools such as Brain connectivity toolbox etc. I do believe there are unique aspects to your package, such as combining information-theoretic measures with network neuroscience to find feature-specific functional networks (I'm not aware of other tools that does this -- I will look it up. If you are aware of others, please include those.
- [ ] **References:** Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper [citation syntax]( https://rmarkdown.rstudio.com/authoring_bibliographies_and_citations.html#citation_syntax)?
Please update references based on additional paper that you will cite to address the above points.
Dear @travisbthomp,
plot_stim_spec_network.py:211: SyntaxWarning: "is" with a literal. Did you mean "=="?
I fixed this morning the warning message by replacing 'is' with '=='
The second figure generated by plot_wf_mi_cc.py does not match the second figure in the corresponding documentation.
Indeed, as you saw, we are mainly using random data for illustrating the functions
The example plot_est_custom.py does not work for me and generates the following error The example plot_ccf.py does not work for me and generates the following error
Indeed, both examples were using the develop branch. This morning I pushed the version 0.4.1 of Frites to Pypi, it should works now once updated.
@whedon generate pdf from branch paper
Attempting PDF compilation from custom branch paper. Reticulating splines etc...
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
After reviewing the paper I would like to make a few comments in the spirit of the review guidelines above.
The description of the high level functionality is not quite clear. "Frites provides such an integrated framework" for what? Is Frites limited to "discovery of cognitive brain networks from multi-channel neurophysiological datasets"? What types of networks? Conversely, I think that the documentation text is quite approachable and nice!
Frites is a Python toolbox for assessing information-based measures on human and animal neurophysiological data (M/EEG, Intracranial). The toolbox also includes directed and undirected connectivity metrics such as group-level statistics on measures of information (information-theory, machine-learning and measures of distance).
In short, the current summary seems obfuscated and potentially difficult to read for a "non-specialist audience".
The paper does indeed have a section titled 'Statement of Need'. However, this section does not establish a clear need for the software nor does it "clearly state what problems the software is designed to solve and who the target audience is". At least, it does not do these things in a way that can be appreciated by an non-expert in the area, such as myself, who is not currently aware of the state of the art approaches, what problems they could solve better, and why Frites fills any of these gaps.
I would suggest revisiting this write up with the journal guidelines in mind.
I cannot find where this has been done in a manner that is decipherable to a non-expert. I would suggest an approach which clearly states the central problem Frites aims to solve and endeavors a comparative discussion to other available tools; what are the advantages of Frites over other packages? If there are no other packages, simply state that Frites is the first of its kind.
The authors do not struggle with the tenets of structural composition. The central message of the statement of need is not currently clear, but this is not an issue with the sentence-by-sentence structure/language/quality per se.
I did not see any issues with the references and all provided DOI links correct redirected to the appropriate paper.
Thank you @madvn and @travisbthomp for the comments on the manuscript, I do agree with both of you about the lack of accessibility. I'll work on the manuscript soon to clarify the text and make it as accessible as possible to the non-expert.
It is not currently clear what primary claims should be assessed. I was able to piece together a possibly-incomplete list from the documentation and accompanying paper.
From the documentation I can see a few things:
From the paper, there are a few more claims to be assessed:
The authors should review this list to make sure that I have not misrepresented, or missed any of, their functionality claims. It seems that verifying these functionality claims would be easiest by pointing to the corresponding example code. To make these claims easier to verify I would suggest considering a strategy similar to the following:
On the main documentation page it might be natural to have a section called something like "What Frites can do for you" between "Description" and "Highlights" which lists the functionality claims and direct link to a corresponding script from the examples section. For instance
"What Frites can do for you"
Essentially, each claim can be made explicit and each claim can be linked to a demonstration of that claim vis-a-vis one of the example scripts.
There are a few performance-related mentions that I can find in the paper. They are as follows (performance-related words are in bold text)
Frites is optimally designed for the analysis of continuous and multi-channel neurophysiological data, encompassing recordings with either uniform spatial sampling (e.g., M/EEG data) and spatially sparse intracranial recordings, such as intracranial EEG or Local Field Potentials (LFPs).
The claim here appears to be that the design of Frites is optimal when considering the task of analyzing particular types of data sets. What does optimal mean here and where can evidence be found for this claim? I did not see, for instance, a documentation example where Frites is compared to other software to show that it is "optimal" amongst the current choices.
Maybe this claim could be made more explicit and confirmed as part of the documentation via a "Performance" page that could be linked off the main bar at the top of the documentation (which currently holds "Overview" "Installation" "API Reference" "Examples")
In order to facilitate automated and efficient usage, Frites provides a set of high-level workflows that integrate several analysis steps from information-based estimation to network level statistical inference.
Similar to above. Where do I find any real information about what workflows really are, let alone how they are automated and enable more efficiency? I assume that would be here but this page only has a cursory discussion of workflows and leaves me with no real concept of what they actually are beyond whatever my intuition suggests from the name "workflow". There is an "in a nutshell" discussion which is not particuarly helpful in addition to a link to "chekout those examples" which takes me to the main examples page so that I must dig to find examples that demonstrate the workflows themselves. Moreover, the links to frites.dataset, frites.workflow and frites.conn on the overview page (above the workflow section) all end in a 404.
The burden of figuring out what a workflow is, and substantiating the claims of automation and efficiency should not be on the reader.
Since several computations implemented in the workflows, such as permutation tests, are computationally demanding, Frites natively supports parallel processing using the Joblib package.
Which computations are the most demanding? Can we point to the examples directly in addition to any performance gains from using Joblib? This would fit well on a "Performance" documentation page as mentioned above
In addition, some functions can further be accelerated using the Numba compiler as an optional dependency. Programming optimizations and external dependencies allow to investigate large-scale datasets in a reasonable time.
What functions can be accelerated / where is acceleration most profitable in Frites? To justify the claim of "large scale datasets in reasonable time" it seems natural to include a benchmark plot showing the processing time of Frites, versus other software packages as well, against a data set size.
@EtienneCmb @travisbthomp @madvn Thanks for your reviews and comments thus far. Could you please update us on the status of your review/responses?
Dear @meg-simula, thank you for updating this thread. I am personally waiting on a response from @EtienneCmb (c.f. my previous remarks) before proceeding further. If I should continue to take some action now please don't hesitate to let me know.
@EtienneCmb We seem to be awaiting your revisions before proceeding with the submission. Could you please give us an update?
@EtienneCmb We still seem to be awaiting your revisions before proceeding with the submission. Could you please give us an update?
Dear Marie,
Sorry for the delay, we are about to submit the revised version and reply to the comments of the reviewers.
Best regards
Dear @meg-simula,
I'm sincerely sorry for the very late answer, I was absorbed in other projects. I'll past out answers to the reviewers today or tomorrow.
Best,
@whedon generate pdf from branch paper
My name is now @editorialbot
Dear @meg-simula and @travisbthomp, please find below our detailed answers to the comments about the paper and about functionality.
In short, the current summary seems obfuscated and potentially difficult to read for a "non-specialist audience".
We agree with the reviewer that the first version of the summary was not accessible to a non-specialist audience. We simplified the section to be closer to the online documentation. In addition, the Summary
section now only includes the context and the problematic that Frites tries to solve.
The paper does indeed have a section titled 'Statement of Need'. However, this section does not establish a clear need for the software nor does it "clearly state what problems the software is designed to solve and who the target audience is". At least, it does not do these things in a way that can be appreciated by an non-expert in the area, such as myself, who is not currently aware of the state of the art approaches, what problems they could solve better, and why Frites fills any of these gaps.
We improved the Statement of need section with the following points :
All of those points are then further detailed in the individual sections.
I cannot find where this has been done in a manner that is decipherable to a non-expert. I would suggest an approach which clearly states the central problem Frites aims to solve and endeavors a comparative discussion to other available tools; what are the advantages of Frites over other packages? If there are no other packages, simply state that Frites is the first of its kind.
Indeed, it was a missing information in the previous manuscript. We now added the sentence : To the best of our knowledge, Frites
is the only Python software combining information-theoretical approaches with flexible group-level analyses for the discovery of cognitive brain networks.
The authors do not struggle with the tenets of structural composition. The central message of the statement of need is not currently clear, but this is not an issue with the sentence-by-sentence structure/language/quality per se.
We hope that this new version clarifies the Statement of need.
On the main documentation page it might be natural to have a section called something like "What Frites can do for you" between "Description" and "Highlights" which lists the functionality claims and direct link to a corresponding script from the examples section.
We thank the reviewer for the in-depth analysis of the functionality of Frites and proposal for amelioration. As suggested, we added a section in the main documentation page entitled “What can you do with Frites?” between "Description" and "Highlights" which lists the functionality claims and direct link to a corresponding overview and script from the examples section.
The claim here appears to be that the design of Frites is optimal when considering the task of analyzing particular types of data sets. What does optimal mean here and where can evidence be found for this claim? I did not see, for instance, a documentation example where Frites is compared to other software to show that it is "optimal" amongst the current choices.
We removed the term “optimal” from the paragraph. Our intention was not to make any claim about optimality, but rather to suitability. The new paragraph now reads as follows:
Frites is suitable for the analysis of continuous and multi-channel neurophysiological data, encompassing recordings with either uniform spatial sampling (e.g., M/EEG data) and spatially sparse intracranial recordings, such as intracranial EEG or Local Field Potentials (LFPs).
Similar to above. Where do I find any real information about what workflows really are, let alone how they are automated and enable more efficiency? I assume that would be here but this page only has a cursory discussion of workflows and leaves me with no real concept of what they actually are beyond whatever my intuition suggests from the name "workflow". There is an "in a nutshell" discussion which is not particularly helpful in addition to a link to "checkout those examples" which takes me to the main examples page so that I must dig to find examples that demonstrate the workflows themselves.
We do agree with the reviewer that the description of the WorkFlows was not sufficient. We now added a clearer description of the workflows in the section Step by step presentation of the workflow.
In addition, we removed any claim of efficient usage from the text to avoid any mis-understanding. In fact, we meant that workflows are higher-level scripts that can be used to run multiple analysis steps.
Moreover, the links to frites.dataset, frites.workflow and frites.conn on the overview page (above the workflow section) all end in a 404.
The dead links on the overview page should now all be working as expected.
Which computations are the most demanding? Can we point to the examples directly in addition to any performance gains from using Joblib? This would fit well on a "Performance" documentation page as mentioned above
In addition, some functions can further be accelerated using the Numba compiler as an optional dependency. Programming optimizations and external dependencies allow to investigate large-scale datasets in a reasonable time.
What functions can be accelerated / where is acceleration most profitable in Frites? To justify the claim of "large scale datasets in reasonable time" it seems natural to include a benchmark plot showing the processing time of Frites, versus other software packages as well, against a data set size.
We thank the reviewer for this great idea of illustrating the performance of Frites. We therefore included a Performance page on the website, in the overview section. This section explains that we are mainly using three tricks to decrease computing time :
We want to thank again @travisbthomp for this detailed review. We hope that those revisions are going to clarify the need for this software, even for non-expert.
@editorialbot generate pdf from branch paper
I'm sorry human, I don't understand that. You can see what commands I support by typing:
@editorialbot commands
@editorialbot set paper as branch
Done! branch is now paper
@editorialbot generate pdf
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
Dear @madvn,
Looks like a very interesting toolbox. I think the idea of feature-specific functional networks using mutual information can be used widely in not just neuroscientific settings but also sociology or even, more topically, epidemiology. I've updated my checklist based on a quick look at the repo and installing the package. I also took a quick look at the paper and have some notes for the authors based on the following requirements while I work my way through the rest of the checklist.
We thank the reviewer for this great remark and for the positive comments. We will try to address the concerns raised.
You have a summary but please revise it to address the "for a diverse, non-specialist audience" part of the requirement. Add a note about the breadth of usability of these tools in other domains besides neuroscience.
We agree that the first version of the “Summary” section was hard to grasp for non-expert readers. We simplified this section and also clarified why measures of information are important for brain data analyses. About the breadth of usability, we added the following sentence to the Measures of information and group-level statistics
section :
While Frites is strongly oriented toward neuroscience applications, the estimators of information are agnostic to data types and can therefore be applied to different fields.
Please place your tool in the space of other information theory tools such as infotheory, IDTXL etc. as well as network neuroscience tools such as Brain connectivity toolbox etc. I do believe there are unique aspects to your package, such as combining information-theoretic measures with network neuroscience to find feature-specific functional networks (I'm not aware of other tools that does this -- I will look it up. If you are aware of others, please include those.
We thank the reviewer for pointing us toward infotheory
and the possible link between Frites and the BCT
. We added the following sentence to the revised manuscript for encompassing those packages :
The networks built from the FC can then be further analyzed using graph-theoretical approaches from the Brain Connectivity Toolbox
(Rubinov and Sporns, 2010). Nevertheless, Frites
is not limited to the included estimators as the definition of custom ones is also supported, such as IDTxl
kernel methods (Wollstadt et al, 2019), infotheory
information decomposition (Candadai & Izquierdo, 2019) or scikit-learn
cross-validated classifiers (Pedregosa et al., 2011). The estimated information can then be combined with the group-level statistics for the identification of feature-specific networks. While Frites
is strongly oriented toward neuroscience applications, the estimators of information are agnostic to data types and can therefore be applied to different fields.
We sincerely thank the reviewer for the feedback,
@editorialbot generate pdf
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
Dear @meg-simula,
Is there anything else we need to do to advance the reviewing process?
Thanks.
Best regards
Dear @EtienneCmb @brovelli (CC: @meg-simula),
Thank you for making changes pursuant to the review of Frites. Regarding advancing the reviewing process (@brovelli): I am currently traveling overseas; I will be happy to review the comments left by @EtienneCmb when I return in a few weeks.
Thank you for your responses @EtienneCmb .. I will get back to you, in the next week.
@EtienneCmb @madvn @travisbthomp Thanks for the updates and responses! Hopefully everyone is back from summer now, and we can get this review sorted out. @madvn and @travisbthomp Could you please have another look at the revised manuscript and software, and concretely, your reviewer checklists above?
@editorialbot generate pdf
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
@EtienneCmb @brovelli @meg-simula
I have completed my review of Frites for this round. I read through the changes and I wanted to thank both @EtienneCmb @brovelli for their work addressing readability issues in addition to severe documentation shortfalls (such as dead links and examples that did not run, etc). I also noted that the authors re-evaluated their performance claims and added a "Performance" documentation page, which is a nice addition since it draws the reader's attention to Joblib and Numba. The reviewers also clarified the current state of the art and Frites' contextual position within this area.
Due to the author's efforts, I have now checked 14 / 19 of the review checklist items. Below, I have detailed each item in the checklist that was checked this round or left unchecked. Each item is marked as Resolved (checked) or includes a To Resolve suggestion. One item is marked Resolved (with a suggestion) indicating that it has been checked but that the authors might consider / reflect upon the suggestion at their discretion.
Review Checklist Items
General Checks
Does this submission meet the scope eligibility described in the JOSS guidelines
Resolved I believe that Frites meets the bar for a substantial scholarly effort.
Functionality
Have the functional claims of the software been confirmed?
Partially. The authors have demonstrated many of the capabilities of the software in their examples section. However, I have raised an issue with one of the two explicitly stated goals of the software. As the functional claim verification occurs in the documentation as part of the examples, I defer this comment to item (5) of the next section
To resolve: See item (5) of the next section.
If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)
Resolved The authors addressed this in their review response. They have removed the performance claims that were brought up in the first round of review.
Documentation
Do the authors clearly state what problems the software is designed to solve and who the target audience is?
The Overview page of the documentation clearly states two goals for Frites. No target audience is stated in the documentation.
To Resolve: Please add a target audience statement to the documentation.
Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
Resolved Yes. The authors provide an installation main page that lists all dependencies and includes instructions for installation with Pip.
Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
Resolved Many examples are included. These are mostly synthetic, but it is understandable that the authors may not have access / permission to release raw data sets. The authors might consider illustrating Frites on some freely available EEG data or similar.
Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
Resolved (with a suggestion) I have marked this as resolved as the authors have included enough examples (in the examples section) to meet the very barest of minimal requirements as the examples do contain the API objects if one digs through them. I do have a suggestion, though. In the "API" section of the documentation you list several API objects/items (Dataset, Workflows, Connectivity, etc). Clicking on these items (e.g. the frites.workflow link) takes me to a separate page. It would be useful to, at the very least, include links to the examples that use each cited API (e.g. WfMi, WfMiCombine, WfConnComod, etc) object on this landing page. In this way, the users can at least see a demonstration of these various API and get some minimal context for how they are used. The API documentation pages are currently as sparse as possible while still being minimally feasible for publication. If the authors want others to actually use their software, post-publication, the API documentation could be made more verbose. This is especially true given the author's explicit claim to be targeting "researchers and students with little programming knowledge".
Are there automated tests or manual steps described so that the functionality of the software can be verified?
The Frites documentation currently ships with several examples. As far as I can see, these currently serve as the "automated tests" that allow for verification of the software functionality. The "Goals" page lists two goals of Frites: (A) Extract task-related cognitive brain networks and (B) Group-level statistical inferences. As a reviewer, I expect the main goals for the functionality to have verifiable examples / resources available. One can argue that (B) is verifiable as there are seven examples of statistical tests in the "Examples" section (for two variables only). However, I cannot find anywhere where a task-related cognitive brain network is extracted / produced. I do see that the authors have included "Autoregressive Models" examples. I looked through these and they do show how Granger causality can be used to infer connectivity but they use synthetic networks a priori. From the documentation goals, one would expect a set of a-priori signal data to be used to create a graph from the GC analysis. I suspect that this would be one of the main tasks that a user of Frites would want to carry out.
To Resolve: To resolve this issue, a clear verification of the author's stated goal (A) seems to be needed. One thing that I can think of would be to create and release a data set (using, e.g. StimSpecAR) of node-wise signals, reflecting say an ~8-10 node graph, and create and example that imports the nodal stimulus data set and then re-creates the connectivity and outputs the weighted graph (or correlation matrix or both) from those signals. The authors may have a better idea.
Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support
I do not see any anything explicit in this direction on the online documentation.
To resolve: Add an explicit mention of this item in line with the review checklist requirements.
Software Paper
Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
The authors revised and simplified the summary section for readability. I think this section is readable by a non-specialist audience. However, this section lacks "a clear description of the high-level functionality and purpose of the software". One could argue that such a description of purpose is provided: (a) implicitly by the final sentence about the "[need for] powerful measures of information to detect complex relations and statistical relevance" (but this is vague) or (b) in the statement of need (but this is not in the summary section, as required by the guidelines)
To resolve: please include "a clear description of the high-level functionality and purpose of the software" in the summary section as requested by the checklist.
Statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
Resolved The statement of need now includes a target audience statement in addition to stating that [Frites is for those] working on the discovery of cognitive brain networks using neurophysiological data. The high-level functionality discussed are: information based analysis, group-level statistical inference, functional connectivity measurement. In light of these observations, I would again cite the need for the authors to demonstrate (e.g. with an example in the documentation) how to "construct a cognitive brain network using neurophysiological data".
Submitting author: !--author-handle-->@EtienneCmb<!--end-author-handle-- (Etienne Combrisson) Repository: https://github.com/brainets/frites Branch with paper.md (empty if default branch): paper Version: v0.4.3 Editor: !--editor-->@meg-simula<!--end-editor-- Reviewers: @madvn, @travisbthomp Archive: 10.5281/zenodo.7278278
:warning: JOSS reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@madvn & @travisbthomp, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @meg-simula know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Review checklist for @madvn
✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
Review checklist for @travisbthomp
✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper