tsalo
commented
2 years ago Based on a conversation with Jonathan Power, here are some ideas about what could be causing the issues:
- Different datasets may exhibit different motion patterns, since motion parameters are influenced by sampling rate.
- This is probably not the case here, because CamCAN and Cambridge have the same TR and are not MB accelerated. They also have very similar mean FD distributions.
- Runs exhibiting more respiration-driven motion may be affecting the results, so it's a good idea to try to exclude these runs from the analyses. Respiration-driven motion is characterized by continuous-looking framewise displacement time series, rather than the more erratic patterns found in purely head motion-driven movement parameters.
- One idea was that median FD would reflect respiration-driven motion more than mean FD, since framewise displacement would always be oscillating around medium levels, rather than staying near zero and spiking with head motions. Therefore, I could look for "humps" in the distribution of median FD, separately for each dataset, to determine appropriate dataset-specific thresholds for the low-motion version of the analyses.
- Outliers might be driving the results. These might be apparent in the distributions of ROI-to-ROI correlation coefficients, for each edge, across runs. The z-transformed coefficients should be normally distributed, so the issue would be identifying outliers in ~35k normally distributed sets of ~700 data points, which isn't a typical multivariate outlier detection issue, since the number of variables is so much higher than the number of data points.
handwerkerd
emdupre
The DDMRA results I get are pretty dissimilar to the original paper's findings. While I expect some differences driven by differences in preprocessing and denoising methods, these go beyond my expectations.
From the replication manuscript, our predictions were:
Analysis of Cambridge, CamCAN, and DuPre datasets
Here are the results using the Cambridge, CamCAN, and DuPre datasets, only dropping subjects with missing or zero-variance data in one or more ROIs (as well as subjects that had already been flagged from other steps). The table shows the p-values for the analyses.
The code used to perform the analysis is this script, and the version of the DDMRA package I used is at https://github.com/tsalo/ddmra/tree/f9be4687d6465c09aae158509566083d456f2567.
Replication on just the Cambridge dataset
Just in case the problem was due to intersite differences (as mentioned in Power et al., 2017), I ran the analyses just using the Cambridge dataset, which should have produced very similar results to the original paper.
Note that the scrubbing analysis results still look terrible, but also the MEDN intercepts were nowhere near significant and the MEDN+GODEC QC:RSFC intercept was significant (and slope was nearly significant).
Replication on subjects with mean FD < 0.2mm
Per Power et al., 2017, DDMRA analyses of multiple datasets can be corrupted by differences in baseline levels of motion, as well as outliers.
Specifically, what it says is:
Also, in the original paper's supplement, they re-run the analyses with only subjects with mean FD < 0.3 and < 0.2 mm, separately, and say that the results were basically the same. I was concerned that the CamCAN dataset, which has higher levels of motion (probably because there are older participants), might be causing problems in the main analysis, so I ran the supplement's analysis of only subjects with < 0.2mm mean FD. However, the results are still quite dissimilar from the original paper.
I don't know if there is a problem with the analyses/dataset or if the CamCAN dataset is simply driving different, but still valid, results.