search

wichmann-lab / python-psignifit

Python implementation of psignifit, for psychometric function estimation
Other
56 stars 25 forks source link

Sometimes the confidence interval's upper bound is lower than the estimate #143

Open lschwetlick opened 5 days ago

lschwetlick commented 5 days ago

See the title.

See here for an example: https://gist.github.com/lschwetlick/b6168ed60668076e0d0c510811c40fd3

This is only the case if you use the percentiles CI method.

otizonaizit commented 5 days ago

@FelixWichmann : do you think this statistically possible and due to the imprecision of estimating the confidence intervals using the percentile method or is it the sign of a bug?

@guillermoaguilar : could you test if matlab shows he same effect using the same data?

FelixWichmann commented 5 days ago

@FelixWichmann : do you think this statistically possible and due to the imprecision of estimating the confidence intervals using the percentile method or is it the sign of a bug? To me rather a sign of a bug, I'm afraid.

@guillermoaguilar : could you test if matlab shows he same effect using the same data? Indeed -- what does MATLAB show?

Sorry but I won't be able to look into psignifit issues until Friday late afternoon/weekend. Please let me know if the problem persists.

Cheers, Felix

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

guillermoaguilar commented 4 days ago

I ran the same data in MATLAB, it does give different results, the estimate is inside the CI. The MATLAB version estimates a much shallower function, and if you look at the marginals they are indeed different between the two implementations.

Here the plots from MATLAB and python (in that order) for the psychometric function:

bug_ci_matlab1 bugci_python1

the marginal for the threshold:

bug_ci_matlab2 bugci_python2

the marginal for the width: bug_ci_matlab3 bugci_python3

Now I have to say there are many things that make this a very special case

And another thought: in this example only the 68% CI does not include the estimate, for 95% and 90% CI it does. It is expected that a 68%CI does not include the estimate in 32% of the time, no? From the definition of the CI.

For me what is more worrying is that the threshold and slope are different, and it seems it comes from how the marginals are computed, as they're different.

guillermoaguilar commented 4 days ago

What do you think @FelixWichmann?

Just for completation here the MATLAB code: example_ci_error.zip

guillermoaguilar commented 4 days ago

And another thought: in this example only the 68% CI does not include the estimate, for 95% and 90% CI it does. It is expected that a 68%CI does not include the estimate in 32% of the time, no? From the definition of the CI.

Nevermind this, I had a Denkfehler. For coverage we calculate how many times the ground truth value is inside the CI. The estimate should be always inside the CI.