Revision Request: Henriques, Rokem, Garyfallidis, St-Jean, Perterson, Correia 2/2

RafaelNH commented 7 years ago

AUTHOR

Dear @ReScience/editors,

I request a review for the following replication:

Original article

Title: Optimization of a free water elimination two-compartment model for diffusion tensor imaging Author(s): Andrew R. Hoy, Cheng G. Koay, Steven R. Kecskemeti, Andrew L. Alexander Journal (or Conference): NeuroImage Year: 2014 DOI: 10.1016/j.neuroimage.2014.09.053 PDF: http://ac.els-cdn.com/S1053811914007952/1-s2.0-S1053811914007952-main.pdf?_tid=7b211902-b7b7-11e6-be78-00000aab0f6b&acdnat=1480591115_87d58e852819d91f683039ddc22f47d3**

Replication

Author(s): Rafael Neto Henriques, Ariel Rokem, Eleftherios Garyfallidis, Samuel St-Jean, Eric Thomas Peterson, Marta Morgado Correia Repository: https://github.com/RafaelNH/ReScience-submission/tree/RNH-AR-EG-SSTJ-ETP-MMC-2016
PDF: https://github.com/RafaelNH/ReScience-submission/blob/RNH-AR-EG-SSTJ-ETP-MMC-2016/article/RNH_AR_EG_SSTJ_ETP_MMC-2016.pdf Keywords: Diffusion MRI; Diffusion modeling; Diffusion tensor imaging; Partial volume; cerebrospinal fluid; free water elimination; partial volume effect Language: English Domain: Life Science

Results

[x] Article has been fully replicated
[ ] Article has been partially replicated
[ ] Article has not been replicated

Potential reviewers

EDITOR

[x] Editor acknowledgment (@pdebuyl) 2 December 2016
[x] Reviewer 1 (@delsuc) 5 December 2016
[x] Reviewer 2 (@soolijoo) 18 December 2016
[x] Review 1 decision [accept] 15 March 2017
[x] Review 2 decision [accept] 30 March 2017
[x] Editor decision [accept] 30 March 2017

RafaelNH commented 7 years ago

Dear @pdebuyl, Due to its merge, I was not able to update PR #25. In this way, I decided to create a new PR to address the comments made by @delsuc and @soolijoo. Below you can find the responses to both reviewers. Regards, @RafaelNH

RafaelNH commented 7 years ago

Response to Reviewer 1

Thanks, @delsuc for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

1.1 General

We agreed that replicating the assessment of the optimal experimental set-up by varying the number of gradient and their intensities might be of interest. Therefore, the replication of figure 2 in the original article and the lower panels of figure 4 have been added to the manuscript. These new analyses evaluate both the optimal set of b-values and the number of b-value shells/directions. In our replication study, we decided to not incorporate Figure 3 of the original article, because this is only related to small adjustments to the two-shell acquisition scheme. We also didn’t incorporate the upper panels of Figure 4, since the larger errors associated to larger f-values are already shown in our article’s Figure 1. Instead, we decided to expand the analysis of the lower panels of Figure 4 to three diffusion measures rather than two, and for two FA levels. The relevant code can be found in the files run_simulations_2.py and run_simulations_3.py of code’s subfolder. Notebook version of these files can be found in the notebook subfolder (run_simulations_2.ipynb and run_simulations_3.ipynb).

1.2 Text

The reason why eq 6 was reordered relative to the original article was added to the manuscript text (see the text right below eq.6): “Relative to the original article, the elements of $W$ have been re-ordered according to the order of the diffusion tensor elements used in Dipy.”
Relatively to the cases of pure free water, we decided to give a deep look into the special case re-initializations proposed by Hoy et al. (2014). Based on the code in supplementary_notebook_3, we observed that the tissue’s mean diffusivity threshold of $1.5 \times 10 ^{-3} mm^{2}s^{-1}$ is more than adequate to identify voxels that contain only water. Therefore, we decided to use also this criterion instead of our initial implementation. With respect to the parameter re-initialization for this cases, some adjustments were done. These are justified in supplementary_notebook_1,ipynb.

1.2 Code

A line of code was added in the notebooks to automatically access the functions directory.
The dependency on Dipy was clarified in article’s subsection “Implementation dependencies”. In this subsection, we also provide some instructions, pointing to Dipy’s installation instructions on the (Dipy website). In addition, our repository includes a README.md file in the code y subfolder, that contains these instructions as well.
Comments about the usage of different Dipy tools were added to all code files and notebooks.

1.3 misc remarks

In run_data a comment was added to stress that 1.7 Gb will be downloaded.
Typos in supplementary_notebook_1 and supplementary_notebook_2 were corrected

RafaelNH commented 7 years ago

Reviewer 2

Thanks @soolijoo for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

2.1 General

@soolijoo pointed out some concerns regarding the theoretical background of the free water DTI (e.g. initial motivation of the use of this technique), the drawback of the simulations based on the same model, and the use of specific parameters such as the number of simulations repetitions and tested SNR levels. We added text to address these concerns, but we stress that we made these choices based primarily on our intention to replicate the work done by Hoy et al., (2014). This motivates the choice of SNR, and other choices of the simulation parameters. Further expansions of the original article go beyond the scope of this replication study.

In addition to the motivation given in the original manuscript, more clarifications on the motivation of the free water DTI model was added in the third paragraph of the manuscript. For further information, we point the reviewer to the following articles which were also added as references of our manuscript:

[8] C Metzler-Baddeley et al. “Frontotemporal connections in episodic memory and aging: A diffusion MRI tractography study”. In: J. Neurosci 31.37 (Sept. 2011), pp. 13236–13245.

[9] L J O’Donnell and O Pasternak. “Does diffusion MRI tell us anything about the white matter? An overview of methods and pitfalls.” In: Schizophr Res. 161.1 (Jan. 2015), pp. 133– 141.

[12] O Pasternak et al. “Hockey Concussion Education Project, Part 2. Microstructural white matter alterations in acutely concussed ice hockey players: a longitudinal free-water MRI study.” In: J Neurosurg. 120.4 (Apr. 2014), pp. 873–881.

2.2 Text

The advantages of using the free water DTI model relative to strategy to suppress free water using less conventional diffusion MRI sequences was added to the fourth paragraph of the manuscript introduction. For further information, we point reviewer to the following article (this article was also added as a reference):

[11] O Pasternak et al. “Free water elimination and mapping from diffusion MRI.” In: Magn. Reson. Med. 62.3 (Sept. 2009), pp. 717–730.

Page 1, Paragraph 2 was reworded to say "For example FA is sensitive to different microstructural properties (such as packing density of axons, and density of myelin in nerve fibers), as well to the degree of white matter coherence (i.e. the alignment of axons within a measurement voxel).”
We also agree that C++ might provide more speed-efficient procedures. Indeed, our main objective here was to provide the first robust open-source reference implementation of fwDTI model. The code was written in Python, to facilitate both its readability (we closely adhere by the PEP8 standard to faciliate) and carefully implemented, including comprehensive test coverage, and revised using Github’s pull request mechanism (https://github.com/nipy/dipy/pull/835). The choice of Python is also further motivated by the large community of potential users, through the Dipy community. While we agree that an even faster implementation could be designed in future work on this topic, this implementation already provides a 20-fold speedup relative to the timing reported by the authors of the original article. Nevertheless, we have rewritten the last paragraph of the introduction to clarify this.
Some authors define Monte-Carlo simulations as any computational algorithms or method that rely on repeated random sampling to obtain numerical results (see for example the definition at Wolfram Mathworld )..Since this term was also adopted in the original article (Hoy et al., 2014), to keep consistency, we decide to continue refer to these simulations as Monte-Carlo simulations.
The justification for the chosen parameters to address the degeneracy of the free-water DTI model when voxel contain mostly water was added in the supplementary notebook 3. There we also address the frequency that this problem arises.
A reference for the chosen free water diffusion value was added in the first paragraph of the methods section. This value is also consistent to what is normally assumed in previous diffusion MRI study including the original article (Hoy et al., 2014).
We agree that an SNR of 20 is more typical in diffusion MRI experiments. We also don’t understand why the original article opted to use this SNR. Here we used this SNR so that figures are replicated in a consistent way relative to the original article. The evaluation of lower SNRs can be seen in the added figure 3. We also rerun Figure 1 for the SNR=20 - apart from larger noise bars, the general metrics bias didn’t change. We decided not to add this as an extra supplementary notebook just for the sake of reducing the number of supplementary notebook, however if interested in also inpecting this please run run_simulations_fig1.py after changing line 55.
Second sentence of the first paragraph of the results section was reworded to: “As reported in the original article, no FA bias are observed for large FA ground truth values and free water volume fractions $f$ ranging around 0 to 0.7 … “

2.3. Code

Diffusion gradient directions were already evenly sampled over the surface of a sphere using an electrostatic potential energy algorithm. This is a standard method used in the field to overcome the pitfall pointed out by the reviewer. Comments were added to the code to clarify this.

2.4. Minor points

Detected Typos were amended
Contrast of in vivo data was adjusted

pdebuyl commented 7 years ago

@RafaelNH Thanks for providing your reply to the referees.

@RafaelNH @arokem @delsuc @soolijoo The review process will continue here.

pdebuyl commented 7 years ago

@delsuc @soolijoo can you acknowledge reception of the revision? This is to make sure that the notification did not get flushed down the inbox/notification. Thank you.

delsuc commented 7 years ago

I acknowledge the reception, I was in congress last week, and I’ll be in vacation (off-grid :-) next week, I’ll try to do my best rapidly

Marc-André

On 13 Feb 2017, at 11:25, Pierre de Buyl notifications@github.com wrote:

@delsuc @soolijoo can you acknowledge reception of the revision? This is to make sure that the notification did not get flushed down the inbox/notification. Thank you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

soolijoo commented 7 years ago

I also acknowledge receipt.

soolijoo commented 7 years ago

Thanks for these comments. I've added my response inline -- to make them easier to find, my additions are prefixed with **

Reviewer 2

Thanks @soolijoo for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

2.1 General

@soolijoo pointed out some concerns regarding the theoretical background of the free water DTI (e.g. initial motivation of the use of this technique), the drawback of the simulations based on the same model, and the use of specific parameters such as the number of simulations repetitions and tested SNR levels. We added text to address these concerns, but we stress that we made these choices based primarily on our intention to replicate the work done by Hoy et al., (2014). This motivates the choice of SNR, and other choices of the simulation parameters. Further expansions of the original article go beyond the scope of this replication study.

In addition to the motivation given in the original manuscript, more clarifications on the motivation of the free water DTI model was added in the third paragraph of the manuscript. For further information, we point the reviewer to the following articles which were also added as references of our manuscript: [8] C Metzler-Baddeley et al. “Frontotemporal connections in episodic memory and aging: A diffusion MRI tractography study”. In: J. Neurosci 31.37 (Sept. 2011), pp. 13236–13245.

[9] L J O’Donnell and O Pasternak. “Does diffusion MRI tell us anything about the white matter? An overview of methods and pitfalls.” In: Schizophr Res. 161.1 (Jan. 2015), pp. 133– 141.

[12] O Pasternak et al. “Hockey Concussion Education Project, Part 2. Microstructural white matter alterations in acutely concussed ice hockey players: a longitudinal free-water MRI study.” In: J Neurosurg. 120.4 (Apr. 2014), pp. 873–881.

** These additions are welcome, but appear to be applications of the same two-tensor based postprocessing approach. My initial query was regarding why postprocessing was the right approach for handling free water rather than eliminating it in the acquisition. Please can you add a small section on why this is appropriate? A few sentences don't seem to be much to ask.

2.2 Text

The advantages of using the free water DTI model relative to strategy to suppress free water using less conventional diffusion MRI sequences was added to the fourth paragraph of the manuscript introduction. For further information, we point reviewer to the following article (this article was also added as a reference): [11] O Pasternak et al. “Free water elimination and mapping from diffusion MRI.” In: Magn. Reson. Med. 62.3 (Sept. 2009), pp. 717–730.

** see above.

Page 1, Paragraph 2 was reworded to say "For example FA is sensitive to different microstructural properties (such as packing density of axons, and density of myelin in nerve fibers), as well to the degree of white matter coherence (i.e. the alignment of axons within a measurement voxel).”

** OK, but it needs to be clear that tesnor data can't be inverted to probvode a unique signiture of these effects -- we see only their net effect.

We also agree that C++ might provide more speed-efficient procedures. Indeed, our main objective here was to provide the first robust open-source reference implementation of fwDTI model. The code was written in Python, to facilitate both its readability (we closely adhere by the PEP8 standard to faciliate) and carefully implemented, including comprehensive test coverage, and revised using Github’s pull request mechanism (nipy/dipy#835). The choice of Python is also further motivated by the large community of potential users, through the Dipy community. While we agree that an even faster implementation could be designed in future work on this topic, this implementation already provides a 20-fold speedup relative to the timing reported by the authors of the original article. Nevertheless, we have rewritten the last paragraph of the introduction to clarify this.

** ok.

Some authors define Monte-Carlo simulations as any computational algorithms or method that rely on repeated random sampling to obtain numerical results (see for example the definition at Wolfram Mathworld )..Since this term was also adopted in the original article (Hoy et al., 2014), to keep consistency, we decide to continue refer to these simulations as Monte-Carlo simulations.

* the link provided is to a definition of the Monte-Carlo method not Monte-Carlo simulation. The method approximates otherwise unfeasible integrals using random processes. a simulation* is an application of the Monte-Carlo method to a given system and as such is a more specialised term. Some authors may indeed confuse these two points, but on my watch. Adding noise to an underlying analytical model is not a simulation, please amend this term.

The justification for the chosen parameters to address the degeneracy of the free-water DTI model when voxel contain mostly water was added in the supplementary notebook 3. There we also address the frequency that this problem arises.

** ok.

A reference for the chosen free water diffusion value was added in the first paragraph of the methods section. This value is also consistent to what is normally assumed in previous diffusion MRI study including the original article (Hoy et al., 2014).

** ok.

We agree that an SNR of 20 is more typical in diffusion MRI experiments. We also don’t understand why the original article opted to use this SNR. Here we used this SNR so that figures are replicated in a consistent way relative to the original article. The evaluation of lower SNRs can be seen in the added figure 3. We also rerun Figure 1 for the SNR=20 - apart from larger noise bars, the general metrics bias didn’t change. We decided not to add this as an extra supplementary notebook just for the sake of reducing the number of supplementary notebook, however if interested in also inpecting this please run run_simulations_fig1.py after changing line 55.

** ok.

Second sentence of the first paragraph of the results section was reworded to: “As reported in the original article, no FA bias are observed for large FA ground truth values and free water volume fractions $f$ ranging around 0 to 0.7 … “

** ok.

2.3. Code

Diffusion gradient directions were already evenly sampled over the surface of a sphere using an electrostatic potential energy algorithm. This is a standard method used in the field to overcome the pitfall pointed out by the reviewer. Comments were added to the code to clarify this.

** my original point was not regarding diffusion gradient directions in the acquisition, it refers to the way in which you are generating random directions in your code. You pick theta and phi from a uniform distribution, which means all generated directions have a bias concentrating them around the poles of your coordinate system. The code needs to be amended such that theta and z are drawn from a uniform distributions, as per the procedure in the original link. Please amend this.

2.4. Minor points

Detected Typos were amended Contrast of in vivo data was adjusted

** great.

Overall recommendation -- accept with minor corrections. Please address the remaining concerns highlighted in this review prior to publication.

pdebuyl commented 7 years ago

Thank you @soolijoo for the report.

About the justification of the methods, do you have more specific concerns? The authors already commented on the existence of non-water-eliminated experimental data, for instance.

pdebuyl commented 7 years ago

@delsuc reminder

soolijoo commented 7 years ago

@pdebuyl I'm still not clear on what clinical or biophysical problem is caused by the the presence of free water without this explicit model. Why is it important to do this, and why is it worth the associated additional scan time and processing? Is there some pathology which cannot be imaged any other way, for example, or is there some widely-made inference which is biased by free water? The existance of free water in the signal is one thing, but why is this important in a given application?

I guess what I'm getting at is the killer reason why I should do this.

delsuc commented 7 years ago

@pdebuyl I'm working on it !

delsuc commented 7 years ago

This version is an important rewrite of the original manuscript, and the text is very much improved.

thanks to there complete set-up, the authors have now been able to performs the simulation for the optimal acquisition conditions. In addition, they have re-estimated of the best procedure for the analysis of voxels containing only free water, which have to be processed apart, and provide what seems to be an increase of the robustness of the method and a substantial improvement of the work itself.

Code is now easier to read, in particular thanks to the comment added, detailing the specific methods of the Dipy module.

All details have been fixed.

One remark, may be, for having the code available in the notebooks, maybe the following syntax :

    import sys
    sys.path.append("../code")

is probably better than

    os.chdir('..//code')

as the authors do.

In conclusion I consider that this manuscript is ready to be published.

pdebuyl commented 7 years ago

Thanks @soolijoo for the clarification.

pdebuyl commented 7 years ago

Thank you @delsuc for the review.

@RafaelNH we are almost there I believe. Let us know when your final update is ready.

Regards, Pierre

RafaelNH commented 7 years ago

Thank you all for your comments.

@delsuc In all notebooks, I changed “os.chdir('..//code')” to what you proposed.

@soolijoo 1) The paragraph 2, 3, 4 and 5 of the article's introduction was further edited to clarify the motivation of the use of the free water DTI model. The main objective of this model is to suppress the partial volume effects of CSF which diffusion coefficient equal to the expected free water value of 3.0e-3 mm2/s due to the absence of tissue barriers. In addition to this, recent studies have shown that measures of free water volume fractions can be also an indication extracellular changes. The use of this model was already shown promising results in removing partial volume confounds in ageing (Metzler-Baddeley, 2012) and to infer extracellular changes in the context of schizophrenia (Pasternak et al., 2012) and brain concussions (Pasternak et al., 2014).

2) I also edited the 2nd paragraph of the article to clarify that DTI is not specific to concrete microstructural changes.

3) Although this was the term used in the original article, I decided to replace the term Monte-Carlos simulations to multi-tensor simulations to avoid further confusions.

4) Regarding the sampling of random directions: we agree with the reviewer that a uniform sampling of directions does result in a bias, with a large concentration around the poles of the coordinate system. Indeed, this issue is well-known in diffusion MRI, and is already addressed in our implementation of the software: our software includes a correction of the initial uniform theta and phi angles. To correct this bias, we use the electrostatic potential energy algorithm (disperse_charges; Jones et al. 1999). For example, see the line of code highlighted in the following image:

directions_code

(from https://github.com/RafaelNH/Free-water-elimination-DTI/blob/master/notebook/run_simulations_2.ipynb).

This was documented in the repository code and notebooks, but we have added a sentence in the methods mentioning the use of this procedure as well as a reference to the Jones et al. (1999) paper describing the algorithm and another explaining the issues in more detail (Jones et al., 2004). For a visual inspection of the performance of the repulsion algorithm please follow the following link: (http://nipy.org/dipy/examples_built/gradients_spheres.html#example-gradients-spheres).

pdebuyl commented 7 years ago

Thank you @RafaelNH for this update.

@delsuc I consider acceptance on your side given your last message and the modification by @RafaelNH

@soolijoo can you confirm whether your last concerns were addressed?

pdebuyl commented 7 years ago

@soolijoo reminder

soolijoo commented 7 years ago

Responses are in bold below.

The paragraph 2, 3, 4 and 5 of the article's introduction was further edited to clarify the motivation of the use of the free water DTI model. The main objective of this model is to suppress the partial volume effects of CSF which diffusion coefficient equal to the expected free water value of 3.0e-3 mm2/s due to the absence of tissue barriers. In addition to this, recent studies have shown that measures of free water volume fractions can be also an indication extracellular changes. The use of this model was already shown promising results in removing partial volume confounds in ageing (Metzler-Baddeley, 2012) and to infer extracellular changes in the context of schizophrenia (Pasternak et al., 2012) and brain concussions (Pasternak et al., 2014).

Ok.

I also edited the 2nd paragraph of the article to clarify that DTI is not specific to concrete microstructural changes. Ok.

Although this was the term used in the original article, I decided to replace the term Monte-Carlos simulations to multi-tensor simulations to avoid further confusions.

Please remove the word "simulation" and replace it with the word "model" throughout.

Regarding the sampling of random directions: we agree with the reviewer that a uniform sampling of directions does result in a bias, with a large concentration around the poles of the coordinate system. Indeed, this issue is well-known in diffusion MRI, and is already addressed in our implementation of the software: our software includes a correction of the initial uniform theta and phi angles. To correct this bias, we use the electrostatic potential energy algorithm (disperse_charges; Jones et al. 1999). For example, see the line of code highlighted in the following image:

(from https://github.com/RafaelNH/Free-water-elimination-DTI/blob/master/notebook/run_simulations_2.ipynb).

This was documented in the repository code and notebooks, but we have added a sentence in the methods mentioning the use of this procedure as well as a reference to the Jones et al. (1999) paper describing the algorithm and another explaining the issues in more detail (Jones et al., 2004). For a visual inspection of the performance of the repulsion algorithm please follow the following link: (http://nipy.org/dipy/examples_built/gradients_spheres.html#example-gradients-spheres).

**This is a very indirect method of doing this -- my original comment included a mathworld link which allows evenly distributed points to be generated simply. Instead of sampling theta and phi unformaly, all that is required is to sample theta and cos(phi) uniformly. What you have done is fine, but dispersing points electrostatically is a more heavyweight operation.

Jones' goal is to construct reproducible sets of points for use as gradient direction sets. For this an electrostatic repulsion method is appropriate since it will lead to approximately even spacing for small numbers of points. You are making much larger sumbers of samples in order to get at the directional statistics of a problem and so a method which is assymtotically even will do.

There's no need to change this, the method is up to you.**

Overall I don't need to see another update, just make these changes and publish.

pdebuyl commented 7 years ago

Hi @soolijoo thank you for this final review.

RafaelNH commented 7 years ago

Thanks @soolijoo for your final review.

@pdebuyl - I've just updated the manuscript with @soolijoo's final comment. The term multi-tensor simulation was replaced to multi-tensor model.

pdebuyl commented 7 years ago

Thanks everyone :-)

@RafaelNH I will start the publication/archival process in the coming days.

pdebuyl commented 7 years ago

@RafaelNH can you provide a few keywords for the article? You can find the keywords for previous ReScience articles here: http://rescience.github.io/read/ Typically they include the main programming language of the reproduction, the general field of research and a few more specific keywords.

pdebuyl commented 7 years ago

Also, can you confirm that the following are the proper last names of all authors? We use this for naming the archived version.

Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia

RafaelNH commented 7 years ago

Keywords: Neuroscience, python, diffusion-weighted imaging, diffusion modeling, diffusion tensor imaging, DTI, free water, partial volume, Cerebrospinal fluid, CSF.

I confirm that the last names are correct!

Many thanks!

pdebuyl commented 7 years ago

Hi @RafaelNH sorry for the delay, I proposed an update to the template that includes an article number, it should make citation a bit easier. Your article is ready for publication, you can check the final pdf here: https://github.com/ReScience-Archives/Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia-2017/tree/master/article

We don't typically do "proofs" in ReScience but why not :-) I am just waiting for your approval of the pdf to publish the final version. Please note that the day of publication is missing on purpose, I'll fill it in.

RafaelNH commented 7 years ago

Hi @pdebuyl! Is the page numeration correct?

pdebuyl commented 7 years ago

Yes. We use since now an article number. Pages are thus 2-1 to 2-10 for your article that is the second article in the volume 3, issue 1, of ReScience.

RafaelNH commented 7 years ago

Ok got it! Is there any way to avoid having Table 2 split in 2 pages? Perhaps inserting this table after the second paragraph of Methods subsection 'Simulations 3' will solve the issue. Apart from this minor, I approve the PDF.

pdebuyl commented 7 years ago

So?

RafaelNH commented 7 years ago

Hi @pdebuyl! Many thanks for the correction! I approve the PDF!

pdebuyl commented 7 years ago

EDITOR

This submission has been accepted for publication and will appear soon at http://rescience.github.io/read/