search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
725 stars 38 forks source link

[REVIEW]: Off-resonance CorrecTion OPen soUrce Software (OCTOPUS) #2578

Closed whedon closed 3 years ago

whedon commented 4 years ago

Submitting author: @imr-framework (Marina Manso Jimeno) Repository: https://github.com/imr-framework/OCTOPUS Version: v1.0.0 Editor: @jni Reviewers: @puolival, @emilljungberg Archive: 10.5281/zenodo.4574444

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/4627d3e6f4f5ba10d9203c7574145e77"><img src="https://joss.theoj.org/papers/4627d3e6f4f5ba10d9203c7574145e77/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/4627d3e6f4f5ba10d9203c7574145e77/status.svg)](https://joss.theoj.org/papers/4627d3e6f4f5ba10d9203c7574145e77)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@puolival, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @jni know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @puolival

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @emilljungberg

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 4 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @puolival it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 4 years ago 
Reference check summary:

OK DOIs

- 10.2217/iim.10.33 is OK
- 10.1109/42.108599 is OK
- 10.1002/1522-2594(200102)45:2<269::AID-MRM1036>3.0.CO;2-5 is OK
- 10.1002/mrm.21599 is OK
- 10.1109/42.781014 is OK
- 10.1109/42.3926 is OK
- 10.1002/mrm.1910250210 is OK
- 10.1002/mrm.1910370523 is OK
- 10.1002/mrm.22428 is OK
- 10.1016/j.mri.2017.07.004 is OK
- 10.1016/j.neuroimage.2011.09.015 is OK
- 10.1109/TMI.2002.808360 is OK
- 10.1109/TSP.2005.853152 is OK
- 10.1016/j.mri.2018.03.008 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

jni commented 4 years ago

@imr-framework @puolival fyi I'm having a bit of trouble finding a second reviewer, so I've decided to start the review early and add the second reviewer manually when I find them.

@puolival thank you again for agreeing to review, please see above for review instructions and a checklist, and please ping me if you have any queries at all!

puolival commented 4 years ago

@jni The URL for accepting the invitation (2. item on reviewer instruction list) doesn't work for some reason. I get the error message "Sorry, we couldn't find that repository invitation. It is possible that the invitation was revoked or that you are not logged into the invited account". (I am logged in so it must be some other reason.)

jni commented 4 years ago

Hmm, @arfon @lpantano any ideas?

jni commented 4 years ago

(@puolival sorry about that! :grimacing: :construction: :grimacing: hold please!)

arfon commented 4 years ago

@whedon re-invite @puolival as reviewer

whedon commented 4 years ago

OK, the reviewer has been re-invited.

@puolival please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

arfon commented 4 years ago

@puolival @jni - not sure what went wrong there. @puolival - could you try clicking the link above again now?

puolival commented 4 years ago

@arfon @jni It worked now and I got a message saying that I was granted push access. Thanks!

jni commented 4 years ago

@puolival this is just a kind and gentle ping, since I don't see any tick marks above. 😊

puolival commented 4 years ago

@jni I am almost finished with the review (1–2 more days), so I thought I would send all comments at once.

jni commented 4 years ago

@whedon add @emilljungberg as reviewer

whedon commented 4 years ago

OK, @emilljungberg is now a reviewer

jni commented 4 years ago

@emilljungberg thank you so much for agreeing to review the paper! At the top of this issue there is a reviewer checklist that you should make your way down. See also the section titled "Reviewer instructions & questions" and links therein. Any questions, just ping me on this issue!

puolival commented 4 years ago

I'll send my comments regarding this submission in two parts. In this first part, I have offered comments regarding the manuscript. In the second part, I will refer to the submission checklist and the GitHub repository.

Comments regarding the software paper

The manuscript needs to be improved before publication. I have offered below a number of comments, which are all intended as constructive criticism:

  1. The abstract should be much expanded. Presently, it doesn’t communicate very well what the paper is about. What are three implemented methods and why were they selected? Who is the target audience of this software (does every MR scientist need it)? What is the present state-of-the-art and broader context of this work? What kind of results can be obtained using the software? Please modify the abstract so that it will be easier for the reader to understand whether the work is relevant for her.

  2. The software uses the NumPy, SciPy, NiBabel, Matplotlib, OpenCV, Pydicom, and PyNUFFT libraries but these have not been cited in the manuscript. Also, the source of the Shepp-Logan phantom image is not stated. The relevant citations should be added.

  3. The reference to Nylund (2014) isn’t complete. Please clarify whether KTH refers to KTH Royal Institute of Technology in Sweden.

  4. In the acknowledgements section, it is stated that funding was received by Fessler but there is no Fessler on the list of authors.

  5. The manuscript should be spell-checked. In the present version, there are typos at least in the summary and statement of need, e.g. ‘Pyton-based’, ‘Resoance’, ‘propreties’, and ‘callibrated’.

  6. What are Google colab notebooks? This is not explained in the manuscript.

  7. Please explain why the simulation is relevant, in a manner that allows non-experts to understand why using a Shepp-Logan head phantom is reasonable. For example, is it simple but allow capturing the most relevant details? This and similar questions now remain unanswered for a broad audience.

  8. Why would researchers want to run the software in a browser? This is stated to be a major benefit. However, to me it is not readily apparent why this is a positive thing. Would it not be better to incorporate the algorithms into existing pipelines, which would allow automation? Please discuss this further.

  9. It is stated that “Furthermore, most of the available packages are also MATLAB-based, restricting the portability, accessibility and customization of the code (Ravi et al., 2018)”. However, MATLAB code can be customized equally well as Python code. It can be also made open-source. Due to these reasons, it is difficult to agree with this statement, unless the authors expand the discussion to pin-point that the previously developed packages are all closed-source. Otherwise I think such a general-level statement isn’t justified in this context.

  10. In the manuscript, we are now given a brief demonstration in which a ground truth is visually compared to three different corrected versions. However, quantitative data should be provided along the visual comparison so that it becomes possible to assess that the software is indeed able to perform the corrections appropriately. This can be reported for example as the increase in signal-to-noise ratio. Related to this, the reader is now also left wondering how much changing the signal-to-noise ratio would change the results. Therefore, I suggest performing the simulations for a range of noise levels. Further, if we take this to a broader level, how useful is the software generally to someone analyzing MR images? What kind of range of scenarios does it cover? Please briefly discuss the limitations of the presented software.

  11. Please describe the functionality of the software in more detail. What is possible and what is not?

Edit: minor fix in one sentence and title.

puolival commented 4 years ago

This is the second part of the review, in which I have referred to the checklist above. I have now ticked in those items of the checklist which I think have been addressed satisfactorily (edit).

Contribution and authorship

The submitting author has contributed to the repository. However, @imr-framework doesn’t appear to belong to any particular person but to a working group. While I think there is no reason to not allow this, I wasn’t able to find a JOSS policy regarding the matter. Therefore, I haven’t yet added a tick mark to this item, to notify the editor to consider how this should be handled.

Substantial scholarly effort

According to the JOSS guidelines, “Reviewers should verify that the software represents substantial scholarly effort. As a rule of thumb, JOSS’ minimum allowable contribution should represent not less than three months of work for an individual.”

Based on the commits made to the repository, for which data is available on the page https://github.com/imr-framework/OCTOPUS/graphs/contributors, this requirement is only met for @Mmj94. The account @imr-framework has 5 commits totaling a net of 738 added lines. However, the majority of these is the GNU license file. @sairamgeethanath has added and removed 7 lines.

Documentation

Some unit tests have been provided but the test source code is mostly not commented. Therefore, I am unfortunately not able confirm that the tests are sufficient to verify the functionality of the software. I suggest the authors to describe what each test does, and what the tests within each file cover overall. Based on the number of tests and available information, it seems that the amount of testing might not yet reach what is expected in the JOSS guidelines.

Software paper

I have posted the comments regarding the manuscript as a separate message; see above.

jni commented 4 years ago

@puolival

this requirement is only met for @Mmj94

This is a collective requirement for the paper as a whole, not meant to be interpreted as needing 3 months of work per author. So I think this means you agree that the requirement is met?

puolival commented 4 years ago

@jni Sorry, I thought it was per person. I do agree that the requirement is met collectively. I have now updated the checklist.

Edit: If possible, I suggest this instruction in the JOSS guidelines to be reworded to be clearer. I was also uncertain if software refers to software + software paper, or only the former.

jni commented 4 years ago

@emilljungberg I see your checklist filling up, and now those issues on the source repo — awesome, thanks for that!

@puolival thank you also for the review above.

@imr-framework @Mmj94 I think at this point we are waiting for some fixes/improvements/responses on your end. Please let us know if you need additional guidance or clarification!

Mmj94 commented 4 years ago

@jni @emilljungberg @puolival Thank you so much for your comments and reviews. Sorry for the delay in my response, I just came back from vacation but I'll definitely start working on all the issues asap.

emilljungberg commented 4 years ago

Review Summary

The OCTOPUS software package consists of python implementations of three different off-resonance correction methods for MRI reconstruction. The package was designed with the intention of making implementation of these methods easier to use, with the motivation that current available software packages, typically in MATLAB, are not flexible enough to accommodate different acquisition strategies. I think the intention behind this work is really good but both the manuscript and the code need improvement before it can fill the gap that the authors have identified.

The python code is well structured into sub-packages and modules with very clear documentation for each function. Many functions have been developed to simplify the interface for the end user, such as im2ksp and ksp2im, as well as generalized data input functions get_data_from_file. However, in doing so, I think the authors end up in the same trap that they tried to avoid, namely not making their tool flexible to all types of data and acquisition strategies. These concerns are highlighted more specifically later in the review.

The demos provided in the OCTOPUS repository clearly demonstrates that the tool is able to correct off-resonance artefacts with the three methods provided. However, it is not clear to me if OCTOPUS is intended to be an end-to-end reconstruction package which includes off-resonance correction, or as a tool for use together with other reconstruction toolboxes. As a complete reconstruction package, OCTOPUS only implements the off-resonance correction and wrappers for 2D-FFT and NUFFT and lacks the advanced reconstruction methods used today such as parallel imaging and compressed sensing.

In the sections I provide comments on the manuscript, the code and some concluding recommendations.

Comments on the manuscript

Summary

Statement of need

Specific comments on writing

Off-resonance is an MR artifact whose source can vary from which can be produced by for instance field inhomogeneities , and to differences in tissue susceptibilities and chemical shift

These phenomena can cause the phase of off-resonant spins (not at the resonant or Larmor frequency) spins to accumulate along the read-out direction.

Users of this toolbox will be familiar with the term off-resonant spins.

Short demo

Specific points to address

Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

It would be good to highlight the three specific methods that this package includes and what the limitations are such as 2D/3D images and Cartesian/non-cartesian trajectories.

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

It is clear that the software corrects off-resonance artefacts but it could be clarified who the software is for and what the typical use of the software would be. Is it intended to mainly operate on reconstructed images or on raw k-space data?

State of the field: Do the authors describe how this software compares to other commonly-used packages?

I think this needs to be clarified in the manuscript. It is stated that available packages are limited and highly specific but without specifying in which way they are specific and how OCTOPUS fills this gap. Considering that off-resonance correction is intended to remove an image artefact, and that it is a computationally heavy problem to solve, comparison to existing packages in terms of image quality and processing time would be useful to evaluate the package.

Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?

The language and writing style in the manuscript is generally of good quality. As mentioned by @puolival, there are a few typos that needs to be checked.

References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

It would be good to extend the reference list with URLs to the available code for the other methods listed in the statement of needs section.

Specific comments on the code

General Checks

Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Based on an assessment of the code for this package, I do not think the current code base satisfies the JOSS scope eligibility criteria. The core of this package is in the ORC module which implements the three off-resonance correction modules in less than 500 lines of code, much of which is doc strings and repeated lines of code for data input. Much of the repository consists of code to demonstrate the use of the toolbox or wrappers to simplify function calls such as FFT or image metric calculations from other packages. Data input/output methods that would be required to make this a powerful package are too simple to be useful, or too specific to the authors use case to be generalisable.

However, I think there is potential for this package to be really useful but it would, in my view, require either extending the package or pivoting in one of a few different ways, which I have outlined in the last section (Suggestions for moving forward)

Functionality

Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

There are no comments on the performance of the code which would have been good to include. A comparison between the different methods would be really useful for users, given that the CPR method is considerable slower than fs-CPR and MFI (~20min for a single slice and coil, compared to about 20s for the two other methods).

Again regarding the use of this tool for image or k-space data, would it work with parallel imaging undersampled data where a direct FFT/NUFFT of the data would result in folding artefacts? I think this needs to be clarified as almost all acquisitions use some type of image acceleration.

Documentation

Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

There is a very nice read the docs API documentation provided for the repository. It would however be good to be more specific regarding the different data structures used. For instance:

Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

The authors have developed unit tests for the core functionality but it could be expanded. In particular it would be good to test the ORC for input data size (1D/2D/3D/4D) and for input data type (magnitude/complex)

Suggestions for moving forward

To make the best use and gain the widest traction of the effort put into this software package, I think it would be good to distinguish between correction of image data and k-space data, as the latter requires additional advanced reconstruction methods.

For an image correction method, the end user would need a robust data input and output method, typically reading either dicom or nifti, together with a field map input or data from which a field map can be reconstructed. For this to be really useful, it would be good to have a python script which can take the input data as commandline argument, choose the correction method, and return a corrected image. That would take this tool to a useful research tool that people could use and modify to their preferences.

For a k-space based correction method on the other hand, the user would need to know how to integrate this tool with other reconstruction toolboxes to produce their final image. A root sum of squares coil combination is rarely sufficient these days and so there needs to be a clear description of how to integrate this method with other reconstruction toolboxes, such as the sigpy framework in python (which I have no affiliation to). To enable this, there would need to be a bit more documentation of the data structures. Also, for processing of raw data, would the methods work with parallel imaging data where a direct FFT of the data results in folding artefacts? For a reconstruction framework, this is a crucial part as almost all acquisitions today use some kind of acceleration.

Overall, the functionality of the methods in OCTOPUS is clear from the examples, but the functionality in terms of compatible input data and how to integrate it into a research framework is not as clear to me. With additional documentation and some development of the code I think this can be resolved.

jni commented 4 years ago

@Mmj94 no pressure from this end, but just want to clarify that we are waiting for input from you at this point, I think! Please let us know if you need any further clarifications, or if you have questions about specific comments.

Mmj94 commented 4 years ago

@jni thanks for the reminder! I have closed some issues in the Github repo and will be addressing other code suggestions/comments with some changes I'll be pushing along the coming week!

Mmj94 commented 4 years ago

Thank you all so much for your patience and again thank you for all your comments and suggestions, I believe they have significantly improved the quality of this tool.

I believe I have addressed all the code comments with my last push to the dev branch. Please have a look at it and let me know if something else needs to be done for acceptance of the paper.

Once the checklists are completed regarding the code, I'll do a pull request to the merge branch and update the ReadTheDocs page, readme, and PyPI package.

There are still two open issues in the repository. I'm hoping Issue #3 will be closed once the manuscript paper is updated. To close Issue #4 I have added some helper functions to check for input dimensions while serving as documentation at the same time, please have a look at it and let me know if further action is needed.

In response to @emilljungberg specific question regarding parallel imaging data, unfortunately at this early stage, OCTOPUS won't be able to correct it.

I'll start addressing the manuscript issues and hopefully will get back to you soon with an update.

jni commented 4 years ago

Thanks for the update @Mmj94! Looking forward to the last few items being resolved and getting this in!

jni commented 4 years ago

Hi all! As per usual, no pressure from me, but just want to make sure things are chugging along and that no one needs help unblocking something!

Mmj94 commented 4 years ago

Hi all, thank you again for your patience during this long process. I believe we've finally implemented all of your comments in both the manuscript and code. Everything is now updated in the repository. For the comments related to the manuscript, I've created a summary document that I'm attaching here.

Please let me know if any other action is required from my side. Thanks again!

Response to reviewers.pdf

puolival commented 4 years ago

@Mmj94 Could you please recompile the manuscript? I'd like to compare the response to reviewers to the updated manuscript. Thanks!

Mmj94 commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

PDF failed to compile for issue #2578 with the following error:

/app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon/author.rb:72:in block in build_affiliation_string': Problem with affiliations for Marina Jimeno, perhaps the affiliations index need quoting? (RuntimeError) from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon/author.rb:71:ineach' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon/author.rb:71:in build_affiliation_string' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon/author.rb:17:ininitialize' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon.rb:205:in new' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon.rb:205:inblock in parse_authors' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon.rb:202:in each' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon.rb:202:inparse_authors' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon.rb:93:in initialize' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon/processor.rb:38:innew' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/lib/whedon/processor.rb:38:in set_paper' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/bin/whedon:58:inprepare' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:ininvoke_command' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/base.rb:466:instart' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-d14a699185fb/bin/whedon:131:in <top (required)>' from /app/vendor/bundle/ruby/2.4.0/bin/whedon:23:inload' from /app/vendor/bundle/ruby/2.4.0/bin/whedon:23:in `

'

Mmj94 commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

jni commented 3 years ago

Hi everyone,

My apologies for the very long silence. It has been a very busy month and I want to thank @puolival and @emilljungberg for the thorough reviews, and @Mmj94 for your response. I need to set aside a chunk of time to consider them all, which unfortunately I can't do until the middle of next week. Apologies that I did not communicate this sooner — I kept thinking that I could do something "in the next few days" and I never found the time. Next week a big deadline passes and I really hope to be able to set some time for this.

Thank you all again. 🙏

emilljungberg commented 3 years ago

Hi,

First of all, thanks for your work on the revisions to the first round of reviews @Mmj94. I think it has really strengthened the manuscript and the toolbox. After looking through the revised manuscript and the repository I have a few points I'd like to highlight though:

In its current state, I think the statements in the paper about OCTOPUS reach beyond its current capacity. The summary reads

OCTOPUS is easy to integrate into other two-dimensional reconstruction pipelines, which makes the tool highly flexible and customizable.

With only being able to use 2D and single-coil data, that is quite a big limitation to me. I think this then comes back in the "Statement of need" section where it is compared to other packages. The paragraph begins with

One can find limited off-resonance correction capabilities within existing packages.

I think limited is the wrong word here, optimised would be better suited as they are developed to be optimised for a certain application, which is understandable as pulse sequences are very different and can benefit from making certain assumptions. Furthermore, I don't think the assessment of the current methods, later on, is fair.

SPIRiT is devoted to correct data acquired using parallel imaging methods, Ostenson’s package only corrects MRF spiral data and implements only one correction method and FUGUE corrects distortion solely on EPI images

Basically all MRI scans today are acquired using multi-channel coils, which would make SPIRiT (to my understanding) applicable to almost all data available out there. MRF scans are highly undersampled acquisitions which understandably would require special considerations. Unless OCTOPUS can be shown to handle all the above situations I think it would be fairer to consider these packages as being optimised rather than limited.

The main point is that the final section then states that OCTOPUS is going to fill this gap in the literature, which to the reader then would imply that it can deal with any acquisition method, thus being able to deal with the type of data for which multiple specialised packages have been developed, as well as data that isn't covered by these specialised methods. While I do think this is a great initiative and I really appreciate accessible software written in Python which we all can learn from, OCTOPUS does not fill this gap in the current state I'm afraid. In essence, my concerns are highlighted in the limitations section which to me are too big to not be implemented in the first version, which includes

  1. 3D data
  2. Multiple receiver coils (Not necessarily being able to deal with aliased data from parallel imaging, but demonstrating that it can deal with multiple receiver coils with different sensitivity profiles).

I'd consider both of these aspects key to making it flexible and practically useable in many situations. 3D imaging is becoming more popular, and multi-channel coils are basically standard today. You mention in the reviewer comments that you see this as a version 0 of the software and that development will be ongoing. Therefore I would suggest further development of the toolbox as planned and then release it in JOSS, in what then might be version 1.0. To my understanding, this wouldn't require a new review process, we can keep this thread going but extend the review process basically.

I also think delaying publication until a more developed version would be a benefit for your team and the software, as you would then showcase a more comprehensive package that will be more applicable to users demands, thus reaching a wider audience.

This is a really great project and I thank you for all your work on it. With further development including 3D data and multi-coil support, which I understand might already be ongoing, I think this will be a really useful tool for researchers and students, and also to learn more about how off-resonance correction in MRI works.

Mmj94 commented 3 years ago

Dear reviewers,

First of all, thank you, @emilljungberg, for your comments and inputs. I totally respect your opinion, but I would like to clarify the multi-coil support limitation you're referring to. OCTOPUS would be able to correct multi-coil data (as long as it's not undersampled) using the current version of the tool. The user could correct each channel data independently and then use a channel combination method of his/her choice. In fact, this is what the example code ORC_main.py does, combining channels using sums of squares after correction. Sorry if the manuscript does not convey that feature correctly and please, let me know if you have further doubts regarding this point.

emilljungberg commented 3 years ago

Thanks @Mmj94 for your rapid reply. Sorry that I missed that reg multiple-coils, now that you mention it I remember commenting on the sum of square coil combination in my first round of comments, my bad. Maybe you could clarify this in the Functionality and Limitations section, that it works with multi-coil data but not parallel imaging acceleration. Out of curiosity, would it work to apply this to coil-combined complex-valued data?

You can consider that point addressed then. Thanks!

Mmj94 commented 3 years ago

Thank you.

Regarding your question, I usually correct each channel independently and then combine them since that's what I've read is most common in the literature. But yes, the tool would work on coil-combined first complex data as well.

jni commented 3 years ago

Hi @Mmj94,

@emilljungberg and I had a private conversation about this, as he felt pretty strongly about the above requirements pre-publication, but wasn't sure about JOSS's editorial policy regarding significant revisions. I'm sorry that things were taking so long. Emil essentially summarised it above — that adding two one features would significantly strengthen the software's position and utility to the community, and thus that publication before that feature is added would be premature. I'm glad that one of Emil's major points is already addressed! 🎉

Generally, my position is that major revisions are a pain in the neck and should not be required for publication — a certain amount of work has been done and the paper reports on that work. But, in this case, I checked with others at the JOSS editorial team and there was consensus that 3D support should be added before publication, given the increasing prevalence of 3D acquisition.

Do you think the 3D data acquisition is something that you could address before publication? On our end at JOSS we'd be happy to wait, as we think it would increase the chance of reuse and community adoption, which is one of the goals we want to pursue here at JOSS.

I'm sorry about this — I have been on the other end of the "major revisions" stick and it sucks and it pains me to write this. But I hope you will see that this would at a minimum be a really cool feature that will mean that lots more people will find your software useful, so that the work is worth doing anyway.

One question I would have about the whole process, though, is whether you in fact have easy access to 3D acquisition data.

Oh and one last thing: one of the JOSS editors flagged that the name Octopus is already used by a tomographic reconstruction framework, which could be confusing. Software name clashes are nothing new and this is not a problem from our end, but something to consider since OCTOPUS is at an early stage and changing the name might not be too traumatic at this point. (Though you do have a cute logo. 😊)

Please don't hesitate to reach out with further questions or concerns about the above.

Mmj94 commented 3 years ago

Hi @jni and @emilljungberg,

First o all, sorry for my delayed response (winter break + deadlines).

We believe that we could do 3D data acquisition if that's what you require for publication. However, could you be a bit more specific about it?

We wondered if you mean 3D multi-slice acquisition, which is already supported, or you mean 3D k-space trajectory acquisition. In the case of the latter, could you provide an example trajectory? We have already used OCTOPUS to correct images from a 3D stack-of-spirals sequence, for example.

Note that for applications where off-resonance is an issue, one would likely use the shortest readout possible for a given trajectory, as off-resonance artifacts worsen with longer durations. For this reason, I don't know if using a 3D k-space trajectory that usually entails longer acquisition times would be relevant. Of course, I'm open to discussion.

jni commented 3 years ago

@emilljungberg I hope all is well with you — I suspect London is not a happy place these days! If you have a chance, would you mind clarifying your request for @Mmj94? Thanks!

emilljungberg commented 3 years ago

Hi again everyone,

Regarding 3D coverage, I was mainly thinking that it would be good to clarify how that could be achieved as it would open up your package to more researchers. If you have done 3D stack of spirals, that's great and shows that it can be done. Any chance you can include an example of those results in the manuscript? So if this capability exists, then maybe you can reword it to something like "Correctio of 3D volumes is possible by processing individual 2D slices (if possible)", as in the case of stack-of-stars/spirals.

Regarding using short readout durations for minimised off-resonance artefacts, I agree with you, shorter readouts are necessary to reduce off-resonance artefacts. But if you reduce the readout for that purpose, you are unlikely to require any corrections. Isn't it in cases where you have long readouts (EPI, spirals) and thus strong artefacts that you actually need these corrections? If your simulation sequence requires B0 variations on the order of kHz, that suggests that such sequence never would require off-resonance correction. And thus, I'm not sure if the simulations you include in the paper are particularly helpful for the average reader.

I appreciate that you had to increase the off-resonance range to see an effect with your short Cartesian readout but that range is way beyond what you ever would observe in an MRI scanner, and thus not a useful reference for someone looking at implementing this for their own use. I understand that it clearly demonstrates that the methods you implemented works as expected, but I think it would be more useful to include a more realistic example. For example, the papers you reference have off-resonance ranges between:

Would you consider changing the sequence used for your simulation to one that would be strongly affected by off-resonance variations on the order of +/- 200 Hz or something like that? Also, if you do have these large B0 variations in the image, I also think you would start to see intra-voxel dephasing due to B0 variation within a single voxel and from that major signal dropout.

If you do want to keep the simulations as is, I would strongly recommend adding in a comment about the large off-resonance variations.

Mmj94 commented 3 years ago

Hi @emilljungberg,

This is an update just to let you know that I'm working on implementing the changes you proposed to the paper's simulations. Specifically:

  1. Single-shot spiral and EPI trajectories for off-resonance ranges of ±100, 150, and 200 Hz.
  2. Try including some results from correction of stack-of-spirals or 3D multi-slice spiral sequences. Note that the results I mentioned earlier about the stack-of-spirals are fMRI images with other types of artifacts apart from off-resonance. For this reason, I don't think I should include those in the manuscript. I'm trying to retrieve some phantom data acquired using that sequence, or there's the possibility I'll have to acquire such data (this may take some extra time).

Would these two items change your mind about the publication of the tool? Of course, I'll refine the manuscript with the changes you suggested too.

Finally, I would like to clarify what I meant with "For this reason, I don't know if using a 3D k-space trajectory that usually entails longer acquisition times would be relevant". I referred to three-dimensional trajectories in k-space, such as 3D cones, which would require excessively long readout durations.

Thank you,

jni commented 3 years ago

@emilljungberg any comment on @Mmj94's questions above?

emilljungberg commented 3 years ago

Hi @Mmj94

Thanks for your comments. I think these sounds like really good additions that will strengthen the paper. With those changes, I'm happy to move forward with publication of the manuscript.

Thank you!

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Mmj94 commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Mmj94 commented 3 years ago

@whedon generate pdf