All points stick to wall

luccaportes commented 3 years ago

Hi, I was doing some tests with my results using two_on_multiple and I got some weird plots.

First, if I set my rope to 0.96 (my results range from 0 to 100) I get this:

However, if I set to 0.97, I get this:

I am using all the default parameters but the "plot" and "rope" of version 1.0.2

Any ideas of why would this happen? Is there any interpretation for this?

luccaportes commented 3 years ago

Here is a code that reproduces it.

from baycomp import two_on_multiple
import pandas as pd
import matplotlib.pyplot as plt

a = [90.89018461774063, 91.05347714737483, 89.86375156582291, 90.11305809025588, 90.15741627962038, 90.50661552134956, 92.59963361038663, 93.54530775534091, 93.42807225800426, 93.6633016834395, 93.40694034499813, 93.58283879907832, 92.838506232675, 93.61307092067871, 88.22167309323994, 88.17771917685411, 88.0645677817309, 88.04713166260647, 87.96398754497321, 88.04551753323445, 88.38568682255541, 78.11311124812975, 93.7353745186701, 75.29755952380953] 
b = [84.76162822381028, 84.59521235293228, 81.95988923189783, 82.1737197260043, 84.15470078404931, 84.92526853206961, 88.5056411625609, 93.48350918283514, 93.26741207268186, 93.34557890060306, 93.286240950892, 93.19325560747559, 92.8103167232695, 93.47227859019856, 89.17796219152201, 89.147564738638, 88.96306960429514, 88.9659003959157, 88.66805711176247, 88.72304391582401, 89.3501235959951, 78.90485894559195, 93.45128273097626, 75.90754464285716]
df = pd.DataFrame([a,b]).transpose()

two_on_multiple(
    df[0], df[1], 
    rope=0.97, plot=True)

plt.show()

benavoli commented 3 years ago

Hi Lucca

Given the Rope and the data, the fact that all posterior samples are "stick to wall" means that the probability C2 is better is zero. Note in fact that p(C1)\approx=0.46 and p(rope)=0.54. This means that the two classifiers are practically equivalent with probability 0.54 and C1 is better than C2 with probability 0.46.

Alessio

janezd / baycomp

All points stick to wall #7