Head-to-head comparison against NORDIC MATLAB implementation?

[tsalo]

Have you had a chance to a do a direct comparison to https://github.com/SteenMoeller/NORDIC_Raw?

[tsalo]

On a related note (probably not worth a separate issue), would you be able/willing to directly connect the parameters in patch-denoising's version of NORDIC to the parameters in NORDIC_Raw?

[paquiteau]

Hi! Yes we did, and we published a paper about it at ISBI 2023: https://hal.science/hal-03895194

The code for reproducing the article lies here: https://github.com/paquiteau/retino-pypeline/

My implementation of NORDIC was probably not good enough, I was not able to reproduced the same results (and I was not able to achieve the same code quality in python /s).

For the mapping of parameters, it should be straightforward; I will do it soon.

[tsalo]

Here are few ideas I had for a head-to-head comparison:

1. Magnitude without noise volumes
2. Magnitude + phase without noise volumes
3. Magnitude with noise volumes
4. Magnitude + phase with noise volumes

I think even running on one run/echo from https://openneuro.org/datasets/ds005250 would be enough for a test.

I assume it would be best to compare each output, but I don't actually have a great idea for how to compare the denoised data from patch-denoise's NORDIC to NORDIC_Raw's NORDIC. Maybe something simple, like making sure the denoised data are more similar to each other than to the un-denoised data? Oh, what about comparing them against MP-PCA denoised data as well? NORDIC_Raw denoised data should be more similar to patch-denoise NORDIC-denoised data than to patch-denoise MP-PCA-denoised data.

Also RMSD between the two g-factor maps. I think the g-factor map is deterministic, so it should be easy to do a direct comparison.

[paquiteau]

I think the comparison could be done on the downstream analysis (where each denoised data undergoes the same t/f-test for instance). Another solution is to look for correlations in times series between the two denoised versions (They should be fairly similar if everything goes well).

The dataset you linked is interesting, but at 2mm-iso (volume size of 110x110x72) with 238 volumes for the first series, I am not sure we would be in a good setup (Noise should be mostly thermal). A fully simulated fMRI run could be more convenient (and we would have more control on the parameters).

Otherwise, I could (probably) give you access to the data we used in our ISBI paper (6 volunteers scanned at 7T with EPI3D, 1mm-iso and TR=2.4s)

[tsalo]

When it comes to regular testing, I prefer to keep the test closer to the actual process, so I like the idea of correlating the denoised time series over running full analyses. From a "this is a useful denoising step" perspective, running the full analyses makes the most sense, but from a "this software works" perspective, just showing that the denoised data look right would be better IMHO.

I agree that a fully simulated run would be better for most of the tests, though testing on real data is also crucial since the assumptions in the simulated data may not always match what happens in real data.

FWIW, I agree that NORDIC probably doesn't do much at 3T (based on previous conversations with Logan Dowdle), but the protocol we used in ds005250 was taken from UMN/WashU collaborators, who have been publishing on how it's super useful (e.g., Moser et al. (preprint)), so... 🤷