SPM v. FDA

[meteor77]

Dear Todd,

I have just run a non-parametric with 10000 permutations in SPM and the same non-parametric procedure with the same permutation on the data and I get different results. Is there an explanation that I can justify this difference? Is it because of smoothing processor or is it something else. It would be a huge help.

Shahab

[0todd0000]

Hi Shahab,

Did you seed the random number generator before running the permutations, like this?

np.random.seed(0)

spm1d's non-parametric procedures use NumPy's built-in random permutation algorithms, so results will generally change if the random number generator is not seeded identically before each run. Nevertheless, 10,000 permutations is usually sufficient to ensure relative stability regardless of seeding. With 10,000 different random permutations I would expect small numerical changes in the critical threshold and p values, but there shouldn't be large, qualitative changes in the results.

Todd

[meteor77]

Hi Todd,

Thanks for your reply. I better rephrase my issue here. The FDA, which I run in MATLAB, gives me a threshold of 4.4 whereas the SPM, which I run in Python or MATLAB gives me a 5.1 threshold. I use non-parametric analysis for both FDA and SPM with a permutation number of 10,000. It seems that the SPM is a bit more conservative when it comes to calculating the threshold. Is there any way that I could justify this?

Shahab

[0todd0000]

Hi Shahab,

Apologies for misunderstanding. As far as I know, FDA uses one-tailed inference. spm1d instead uses two-tailed inference by default. If you used two-tailed inference in spm1d, please try re-running with one-tailed inference and re-comparing to the FDA result.

If that doesn't solve the problem we may need to look at the actual data to figure out what is causing the discrepancy.

Todd

[meteor77]

Hi Todd,

It seems like every time that I repeat the same test with the same inputs in SPM I would get a different threshold value every time. Now I am really confused.

[0todd0000]

Dear Shahab,

Are you controlling the random number generator state?

Please see the example scripts in: ./spm1d/examples/nonparam/

All of these scripts control randomization (in this case: random permutation selection) via the functions:

(Python)

np.random.seed(0)

(MATLAB)

rng(0)

If you control the random number generator state, the results should be precisely reproducible because the (randomly selected) permutations will be precisely reproduced.

Todd

[meteor77]

Hi Todd

That was quick. Thanks. I think I may know where the difference lays between FDA and SPM non-parametric procedure. The FDA doesn't have a function to control the random number generator state (to my knowledge at least (please correct me if I am wrong)). This causes a fluctuation in producing threshold values each time I repeat the test in FDA which I think that the results could be unreliable in that aspect.

Many thanks for the help.

Shahab

[0todd0000]

Hi Shahab,

That sounds quite plausible. However, I am not familiar with all details of the FDA software package, so if the fluctuation you have observed occurs in the FDA results, then please contact FDA developers for feedback.

Fluctuating results, whether in FDA or in spm1d, do not necessarily imply that the procedures are unreliable. They could simply imply an insufficient number of permutations. If the design is simple and if the data are beautifully normal then 10,000 iterations should be sufficient to ensure numerical stability. With complex designs and/or with departures from normality, 1,000,000 or more iterations might be necessary to produce probabilistic convergence, depending on the underlying distribution(s) and/or the probability-relevant question.

Just to confirm: do you see similar fluctuations in spm1d results? As far as I know they should be precisely repeatable if you control the built-in random number generator state.

Todd

[meteor77]

Hi Todd,

Yes. After controlling the random number generator state, I am able to reproduce the same values.

Thanks again for your help.

Shahab