p-value

[peoplecure]

Hi,

While reproducing some of the results, I have come across some incongruity in the p-values.

In the paper, it was reported that WEAT1 has a p-value of 10^−7. However, the provided code was showing up as p = 0. So, I looked at the weat_perm and noticed that p_val <- sum(reps > S_xyab)/length(reps)

So, I've added this code on the original weat_perm function:

  reps[i] <- sum(S_xab) - sum(S_yab)
  # reps' object of i is the subtraction of those two

  }
  print(reps[i])
  print(S_xyab)
  print(sum(reps > S_xyab))
  print(length(reps))

  p_val <- sum(reps > S_xyab)/length(reps) 

which returned

[1] -0.4718922
[1] 2.238165
[1] 0
[1] 1000
    S_xyab        d p_value
1 2.238165 1.504315       0

I take it sum(reps > S_xyab) most likely returns 0, thus the 0 p-value. But, how was the p-value obtained in the original study? Did you use a different method for the paper publication?

Many thanks.

[conjugateprior]

Because this is a permutation p-value constructed by simulation when the 'true' one-sided p-value is very small you'll need a lot of runs to get a single element of reps to be larger than S_xyab. This should eventually happen if you make the function parameter b large enough.

In the paper, the authors ran huge numbers of permutations. I have not made the default as generally I feel like the difference between say 0.0000001 and 0.00000001 in problems like these is of essentially no inferential relevance. We just learned that S_xyab is very small under the null.

[peoplecure]

Thank you for the response. I needed the method to show the replicated results in my paper. I will try it over the weekend. : )