[REVIEW]: The Neuroimaging Data Model (NIDM) Linear Regression Tool

Submitting author: @AKUMAR0019 (Ashmita Kumar) Repository: https://github.com/incf-nidash/PyNIDM Version: v.3.8.2 Editor: @osorensen Reviewer: @htwangtw, @robbisg Archive: Pending

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/194b71c9943d2550be0df98ce62373df"><img src="https://joss.theoj.org/papers/194b71c9943d2550be0df98ce62373df/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/194b71c9943d2550be0df98ce62373df/status.svg)](https://joss.theoj.org/papers/194b71c9943d2550be0df98ce62373df)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@htwangtw & @robbisg, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @osorensen know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @htwangtw

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[ ] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@AKUMAR0019) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @robbisg

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@AKUMAR0019) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[ ] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

@AKUMAR0019 @robbisg @htwangtw; after giving it some thought, I've decided to add a query-scope label on this submission. I share the concern stated by @robbisg and @htwangtw in their posts just above that the contribution of this submission might be too small to be of sufficient scholarly effort for a JOSS publication. In particular, I am not sure if the submission satisfies the following criteria:

“Minor utility” packages, including “thin” API clients, and single-function packages are not acceptable.

@openjournals/joss-eics; please note that the submission is an extension of an existing tool, and is mainly this file, if I have understood correctly.

@osorensen The linear regression tool uses the REST-API to query NIDM files which is located through this link (https://github.com/incf-nidash/PyNIDM/blob/master/nidm/experiment/tools/rest.py) and should be added to the lines of code in the linear regression tool to get an accurate estimate. In our interpretation of the JOSS guidelines, the total lines of code exceeds the journal minimum. Further, one can find the PyNIDM code included in the source file for additional reference to code written as part of this submission: “from nidm.experiment.Query import GetProjectsUUID”, “from nidm.experiment.tools.rest import RestParser”.

The scholarly effort is the development of a tool capable of learning linear models in the context of linked-data, RDF documents for human neuroimaging-related neuroscience (i.e. NIDM). There are no other tools supporting such statistical investigation of human neuroimaging-related neuroscience data/metadata stored in RDF documents that we know of. The Neuroimaging Data Model (NIDM) was started by an international team of cognitive scientists, computer scientists and statisticians to develop a data format capable of describing all aspects of the data lifecycle, from raw data through analyses and provenance. The NIDM working group officially began meeting in 2011 focusing on improving how data, analyses, and provenance are shared in the neuroimaging community. The NIDM group participates in monthly teleconference meetings, with 9-12 attendees per month on average, representing 12 institutions across Europe and the US. Over eighteen abstracts have been published at international meetings as well as 3 journal publications describing the group’s work. All NIDM group meeting minutes and twenty code repositories are publicly available on GoogleDocs and GitHub, respectively, with a total of over 215 meeting records, demonstrating the group's commitment to open, community-based, development and evidencing the scholarly effort of this work.

This work is not an extension of an existing tool. It is a new tool built on top of the NIDM work which spans a wide range of functionality to track the lifecycle of data from acquisition to results. This particular tool was developed along with the associated query functionality, linked above, to make running linear models on linked-data NIDM graphs possible across different datasets. There is no other tool that provides such functionality.

Thanks a lot for your explanation, @AKUMAR0019. The editorial team will take this into consideration when evaluating the scope query. You can expect to hear back from us in less than two weeks.

Dear @AKUMAR0019, I have now discussed with the editorial team, and I am sorry to inform you that we have concluded that this submission currently does not satisfy JOSS's requirement of substantial scholarly effort.

However, we also see that both PyNIDM and features like this linear regression tool are highly valuable, and therefore invite you to make a major revision of this submission. We will hence pause this review for now, and welcome you back once the following points are addressed.

Scope The linear regression tool in its current state is not of sufficient scope for JOSS. In your response you point to the RESTparse, but this seems to be a multi-purpose tool that is used throughout NIDM experiments more broadly. If this is part of the submission, it should be made clear in the paper. Currently I cannot see that this tool is mentioned.

We have some suggestions for increasing the scope:

As PyNIDM itself does not seem to be published, you are welcome to submit PyNIDM or sufficient parts of it. If submitting the whole tool, the submission should explicitly provide updates from the previously published nidm-rest and nidm-query tools.
Alternatively, the code base of the linear regression could be significantly extended.

Points raised by reviewers In addition, the points raised by the reviewers above should be adequately addressed. In particular, the paper should convince the reader about the utility of this tool, the quality of the graphics could be improved, and the references properly formatted.

We understand that this may disappointing, but hope that you wish to make a major revision. If so, once you're ready please tag me in this thread, and I'll remove the pause label.

@robbisg and @htwangtw thank you very much for your thorough reviews so far. I will contact you again later if the authors choose to make a major revision.

Thanks for the thoughtful feedback on the submission @htwangtw, @robbisg, and @osorensen.

@AKUMAR0019 – our preferred process in situations like this is to reject the submission and invite a resubmission in the future if/when there have been substantial changes that meet the feedback from the first review.

As such, I'm going to go ahead and reject the submission now.

@arfon @osorensen @robbisg @htwangtw We are sorry to hear of JOSS’s final decision. Upon initial review these concerns should have been brought up and an initial decision (minor revisions, major revisions, or rejection) made. Instead we received ad-hoc comments and requests for changes which we made. After a couple of rounds this paper was ultimately rejected. This process is unlike any scientific journal we have published in and is a process we strongly encourage JOSS to change as it wastes both ours and your time. If the original paper did not meet the JOSS guidelines then it should have been rejected straight away and we would have used our valuable time to submit it to another journal. Please consider revising your review methodology.

@AKUMAR0019 – I'm sorry that you are disappointed in the review experience here. JOSS reviews by design are interactive, conversational, and iterative. We consider this a feature of our review experience, and it has stood us in good stead over the past 5 years and nearly 1500 published papers.

This submission was a little more complex than some for us as your tool is a small part of a much larger piece of software. As such, the concern around the "Substantial scholarly effort" criterion only became apparent well into the review process which while later than most submissions, is still a valid time for this to be raised. Overall, I believe the reviewers and editor performed their duties very well here and I thank them for their time.

openjournals / joss-reviews