rahlk commented 8 years ago

[~]$ python -B ~/git/rahlk/Bellwether/src/TCA+/par_exec.py

Summary (Now compiling....)

Looked at Pd, Pf, G (where g = sqrt(Pd*(1-Pf)); the harmonic mean between Pd and Pf used for sorting)
No clear Bellwether on any of the metrics. We were able to find on a metric called Balance (which was the euclidean distance from ideal score (Pd, Pf) = (1, 0)). Our results here.
Now, note that out round robin evaluation strategy of finding the best dataset significantly outperforms TCA+
The authors of TCA+ do not report results on this data set. Moreover, they conveniently breeze through the most important part, the TCA algorithm. See sections 3.1 and 3.2 (this was a beast to implement! But offered no major benefits).
Their primary contribution of TCA+ is a proposal on choosing the best normalization strategy.

Comparisons

(I'm sorry, this is a formatted raw dump. I'll create a summary table and paste it here.)

Target Project: ANT

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.60 | 0.60 | 0.49 |
+----------+------+------+------+
| poi      | 0.59 | 0.60 | 0.48 |
+----------+------+------+------+
| velocity | 0.74 | 0.69 | 0.47 |
+----------+------+------+------+
| log4j    | 0.69 | 0.69 | 0.46 |
+----------+------+------+------+
| xerces   | 0.86 | 0.83 | 0.38 |
+----------+------+------+------+
| camel    | 0.08 | 0.06 | 0.27 |
+----------+------+------+------+
| jedit    | 0.06 | 0.09 | 0.23 |
+----------+------+------+------+
| ivy      | 0.05 | 0.07 | 0.22 |
+----------+------+------+------+
| xalan    | 0.03 | 0.03 | 0.17 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| ivy      | 0.67 | 0.25 | 0.71 |
+----------+------+------+------+
| jedit    | 0.61 | 0.20 | 0.70 |
+----------+------+------+------+
| xalan    | 0.72 | 0.32 | 0.70 |
+----------+------+------+------+
| lucene   | 0.77 | 0.39 | 0.69 |
+----------+------+------+------+
| poi      | 0.78 | 0.40 | 0.68 |
+----------+------+------+------+
| camel    | 0.79 | 0.45 | 0.66 |
+----------+------+------+------+
| velocity | 0.65 | 0.34 | 0.66 |
+----------+------+------+------+
| log4j    | 0.89 | 0.54 | 0.64 |
+----------+------+------+------+
| xerces   | 0.83 | 0.53 | 0.63 |
+----------+------+------+------+

Target Project: LUCENE

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.76 | 0.78 | 0.41 |
+----------+------+------+------+
| poi      | 0.28 | 0.47 | 0.39 |
+----------+------+------+------+
| log4j    | 0.92 | 0.90 | 0.30 |
+----------+------+------+------+
| xerces   | 0.09 | 0.07 | 0.29 |
+----------+------+------+------+
| ant      | 0.05 | 0.08 | 0.22 |
+----------+------+------+------+
| camel    | 0.05 | 0.03 | 0.22 |
+----------+------+------+------+
| jedit    | 0.03 | 0.11 | 0.17 |
+----------+------+------+------+
| ivy      | 0.03 | 0.03 | 0.17 |
+----------+------+------+------+
| xalan    | 0.03 | 0.05 | 0.15 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| camel    | 0.58 | 0.27 | 0.65 |
+----------+------+------+------+
| log4j    | 0.75 | 0.46 | 0.63 |
+----------+------+------+------+
| poi      | 0.41 | 0.17 | 0.58 |
+----------+------+------+------+
| ivy      | 0.39 | 0.17 | 0.57 |
+----------+------+------+------+
| xerces   | 0.62 | 0.47 | 0.57 |
+----------+------+------+------+
| velocity | 0.42 | 0.26 | 0.56 |
+----------+------+------+------+
| xalan    | 0.42 | 0.29 | 0.55 |
+----------+------+------+------+
| ant      | 0.32 | 0.12 | 0.53 |
+----------+------+------+------+
| jedit    | 0.22 | 0.06 | 0.45 |
+----------+------+------+------+

Target Project: JEDIT

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.48 | 0.46 | 0.51 |
+----------+------+------+------+
| poi      | 0.53 | 0.55 | 0.49 |
+----------+------+------+------+
| lucene   | 0.42 | 0.44 | 0.48 |
+----------+------+------+------+
| xerces   | 0.81 | 0.80 | 0.41 |
+----------+------+------+------+
| xalan    | 0.12 | 0.14 | 0.32 |
+----------+------+------+------+
| ivy      | 0.10 | 0.12 | 0.30 |
+----------+------+------+------+
| log4j    | 0.92 | 0.93 | 0.26 |
+----------+------+------+------+
| ant      | 0.04 | 0.03 | 0.19 |
+----------+------+------+------+
| camel    | 0.00 | 0.03 | 0.06 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| ant      | 0.68 | 0.26 | 0.71 |
+----------+------+------+------+
| xalan    | 0.65 | 0.33 | 0.66 |
+----------+------+------+------+
| ivy      | 0.51 | 0.17 | 0.65 |
+----------+------+------+------+
| lucene   | 0.67 | 0.36 | 0.65 |
+----------+------+------+------+
| camel    | 0.76 | 0.47 | 0.63 |
+----------+------+------+------+
| poi      | 0.48 | 0.23 | 0.61 |
+----------+------+------+------+
| velocity | 0.62 | 0.44 | 0.59 |
+----------+------+------+------+
| log4j    | 0.91 | 0.64 | 0.57 |
+----------+------+------+------+
| xerces   | 0.80 | 0.61 | 0.56 |
+----------+------+------+------+

Target Project: XERCES

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.52 | 0.56 | 0.48 |
+----------+------+------+------+
| lucene   | 0.37 | 0.41 | 0.47 |
+----------+------+------+------+
| poi      | 0.30 | 0.31 | 0.46 |
+----------+------+------+------+
| xalan    | 0.24 | 0.27 | 0.42 |
+----------+------+------+------+
| log4j    | 0.74 | 0.85 | 0.34 |
+----------+------+------+------+
| jedit    | 0.03 | 0.02 | 0.17 |
+----------+------+------+------+
| ivy      | 0.02 | 0.02 | 0.15 |
+----------+------+------+------+
| camel    | 0.02 | 0.01 | 0.13 |
+----------+------+------+------+
| ant      | 0.02 | 0.01 | 0.12 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| log4j    | 0.47 | 0.27 | 0.59 |
+----------+------+------+------+
| camel    | 0.44 | 0.32 | 0.55 |
+----------+------+------+------+
| velocity | 0.41 | 0.26 | 0.55 |
+----------+------+------+------+
| xalan    | 0.41 | 0.31 | 0.53 |
+----------+------+------+------+
| lucene   | 0.38 | 0.33 | 0.50 |
+----------+------+------+------+
| poi      | 0.31 | 0.21 | 0.50 |
+----------+------+------+------+
| ant      | 0.17 | 0.09 | 0.39 |
+----------+------+------+------+
| jedit    | 0.16 | 0.07 | 0.38 |
+----------+------+------+------+
| ivy      | 0.14 | 0.07 | 0.36 |
+----------+------+------+------+

Target Project: XALAN

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| log4j    | 0.47 | 0.37 | 0.55 |
+----------+------+------+------+
| lucene   | 0.35 | 0.40 | 0.46 |
+----------+------+------+------+
| xerces   | 0.70 | 0.70 | 0.46 |
+----------+------+------+------+
| poi      | 0.33 | 0.35 | 0.46 |
+----------+------+------+------+
| velocity | 0.26 | 0.26 | 0.44 |
+----------+------+------+------+
| camel    | 0.11 | 0.10 | 0.32 |
+----------+------+------+------+
| ivy      | 0.10 | 0.13 | 0.30 |
+----------+------+------+------+
| jedit    | 0.07 | 0.03 | 0.26 |
+----------+------+------+------+
| ant      | 0.06 | 0.05 | 0.24 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.55 | 0.42 | 0.57 |
+----------+------+------+------+
| camel    | 0.49 | 0.37 | 0.56 |
+----------+------+------+------+
| log4j    | 0.55 | 0.43 | 0.56 |
+----------+------+------+------+
| xerces   | 0.62 | 0.55 | 0.53 |
+----------+------+------+------+
| velocity | 0.34 | 0.26 | 0.50 |
+----------+------+------+------+
| ant      | 0.27 | 0.14 | 0.49 |
+----------+------+------+------+
| poi      | 0.29 | 0.19 | 0.48 |
+----------+------+------+------+
| ivy      | 0.25 | 0.15 | 0.46 |
+----------+------+------+------+
| jedit    | 0.23 | 0.13 | 0.45 |
+----------+------+------+------+

Target Project: CAMEL

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.74 | 0.62 | 0.53 |
+----------+------+------+------+
| log4j    | 0.55 | 0.55 | 0.50 |
+----------+------+------+------+
| xalan    | 0.59 | 0.61 | 0.48 |
+----------+------+------+------+
| lucene   | 0.31 | 0.35 | 0.45 |
+----------+------+------+------+
| poi      | 0.25 | 0.24 | 0.44 |
+----------+------+------+------+
| xerces   | 0.86 | 0.85 | 0.36 |
+----------+------+------+------+
| jedit    | 0.01 | 0.01 | 0.12 |
+----------+------+------+------+
| ant      | 0.01 | 0.01 | 0.08 |
+----------+------+------+------+
| ivy      | 0.01 | 0.02 | 0.08 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.51 | 0.33 | 0.59 |
+----------+------+------+------+
| xalan    | 0.44 | 0.28 | 0.57 |
+----------+------+------+------+
| log4j    | 0.67 | 0.53 | 0.56 |
+----------+------+------+------+
| velocity | 0.56 | 0.44 | 0.56 |
+----------+------+------+------+
| xerces   | 0.60 | 0.54 | 0.53 |
+----------+------+------+------+
| ant      | 0.28 | 0.14 | 0.49 |
+----------+------+------+------+
| poi      | 0.29 | 0.20 | 0.49 |
+----------+------+------+------+
| ivy      | 0.26 | 0.17 | 0.46 |
+----------+------+------+------+
| jedit    | 0.18 | 0.07 | 0.41 |
+----------+------+------+------+

Target Project: VELOCITY

TCA+

+--------+------+------+------+
|  Name  |  Pd  |  Pf  |  G   |
+========+======+======+======+
| lucene | 0.54 | 0.56 | 0.49 |
+--------+------+------+------+
| poi    | 0.29 | 0.25 | 0.46 |
+--------+------+------+------+
| camel  | 0.23 | 0.24 | 0.41 |
+--------+------+------+------+
| log4j  | 0.77 | 0.80 | 0.39 |
+--------+------+------+------+
| xerces | 0.04 | 0.08 | 0.20 |
+--------+------+------+------+
| ivy    | 0.03 | 0.02 | 0.18 |
+--------+------+------+------+
| ant    | 0.02 | 0.07 | 0.13 |
+--------+------+------+------+
| jedit  | 0.01 | 0.01 | 0.12 |
+--------+------+------+------+
| xalan  | 0.01 | 0.03 | 0.10 |
+--------+------+------+------+

Bellwether Method

+--------+------+------+------+
|  Name  |  Pd  |  Pf  |  G   |
+========+======+======+======+
| lucene | 0.52 | 0.35 | 0.58 |
+--------+------+------+------+
| xalan  | 0.47 | 0.29 | 0.58 |
+--------+------+------+------+
| log4j  | 0.77 | 0.57 | 0.57 |
+--------+------+------+------+
| camel  | 0.43 | 0.39 | 0.51 |
+--------+------+------+------+
| xerces | 0.47 | 0.44 | 0.51 |
+--------+------+------+------+
| ant    | 0.29 | 0.15 | 0.49 |
+--------+------+------+------+
| poi    | 0.31 | 0.24 | 0.49 |
+--------+------+------+------+
| ivy    | 0.19 | 0.09 | 0.42 |
+--------+------+------+------+
| jedit  | 0.11 | 0.07 | 0.31 |
+--------+------+------+------+

Target Project: POI

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.56 | 0.55 | 0.50 |
+----------+------+------+------+
| lucene   | 0.82 | 0.78 | 0.43 |
+----------+------+------+------+
| log4j    | 0.89 | 0.86 | 0.35 |
+----------+------+------+------+
| xerces   | 0.09 | 0.12 | 0.29 |
+----------+------+------+------+
| xalan    | 0.07 | 0.14 | 0.25 |
+----------+------+------+------+
| ivy      | 0.04 | 0.03 | 0.21 |
+----------+------+------+------+
| ant      | 0.02 | 0.01 | 0.14 |
+----------+------+------+------+
| jedit    | 0.02 | 0.02 | 0.12 |
+----------+------+------+------+
| camel    | 0.01 | 0.00 | 0.08 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.73 | 0.40 | 0.66 |
+----------+------+------+------+
| camel    | 0.75 | 0.51 | 0.61 |
+----------+------+------+------+
| ivy      | 0.44 | 0.19 | 0.60 |
+----------+------+------+------+
| log4j    | 0.85 | 0.59 | 0.59 |
+----------+------+------+------+
| xalan    | 0.50 | 0.33 | 0.58 |
+----------+------+------+------+
| xerces   | 0.42 | 0.34 | 0.53 |
+----------+------+------+------+
| ant      | 0.27 | 0.13 | 0.48 |
+----------+------+------+------+
| velocity | 0.22 | 0.27 | 0.40 |
+----------+------+------+------+
| jedit    | 0.14 | 0.05 | 0.36 |
+----------+------+------+------+

Target Project: LOG4J

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.46 | 0.32 | 0.56 |
+----------+------+------+------+
| xerces   | 0.31 | 0.32 | 0.46 |
+----------+------+------+------+
| poi      | 0.67 | 0.69 | 0.45 |
+----------+------+------+------+
| velocity | 0.86 | 0.89 | 0.30 |
+----------+------+------+------+
| ivy      | 0.11 | 0.18 | 0.30 |
+----------+------+------+------+
| jedit    | 0.07 | 0.04 | 0.25 |
+----------+------+------+------+
| ant      | 0.03 | 0.03 | 0.18 |
+----------+------+------+------+
| xalan    | 0.02 | 0.03 | 0.15 |
+----------+------+------+------+
| camel    | 0.02 | 0.01 | 0.15 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| camel    | 0.52 | 0.21 | 0.64 |
+----------+------+------+------+
| poi      | 0.38 | 0.10 | 0.59 |
+----------+------+------+------+
| velocity | 0.45 | 0.21 | 0.59 |
+----------+------+------+------+
| lucene   | 0.36 | 0.08 | 0.57 |
+----------+------+------+------+
| xalan    | 0.50 | 0.34 | 0.57 |
+----------+------+------+------+
| ant      | 0.28 | 0.03 | 0.52 |
+----------+------+------+------+
| xerces   | 0.66 | 0.59 | 0.52 |
+----------+------+------+------+
| ivy      | 0.27 | 0.03 | 0.51 |
+----------+------+------+------+
| jedit    | 0.19 | 0.02 | 0.43 |
+----------+------+------+------+

Target Project: IVY

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| jedit    | 0.39 | 0.22 | 0.55 |
+----------+------+------+------+
| poi      | 0.30 | 0.19 | 0.50 |
+----------+------+------+------+
| lucene   | 0.85 | 0.76 | 0.45 |
+----------+------+------+------+
| velocity | 0.72 | 0.73 | 0.44 |
+----------+------+------+------+
| log4j    | 0.62 | 0.70 | 0.43 |
+----------+------+------+------+
| ant      | 0.10 | 0.07 | 0.31 |
+----------+------+------+------+
| camel    | 0.08 | 0.09 | 0.26 |
+----------+------+------+------+
| xerces   | 0.06 | 0.07 | 0.23 |
+----------+------+------+------+
| xalan    | 0.02 | 0.06 | 0.13 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.71 | 0.38 | 0.66 |
+----------+------+------+------+
| ant      | 0.49 | 0.16 | 0.64 |
+----------+------+------+------+
| camel    | 0.78 | 0.47 | 0.64 |
+----------+------+------+------+
| log4j    | 0.84 | 0.51 | 0.64 |
+----------+------+------+------+
| poi      | 0.53 | 0.24 | 0.64 |
+----------+------+------+------+
| xalan    | 0.57 | 0.29 | 0.64 |
+----------+------+------+------+
| velocity | 0.69 | 0.46 | 0.61 |
+----------+------+------+------+
| jedit    | 0.39 | 0.19 | 0.57 |
+----------+------+------+------+
| xerces   | 0.75 | 0.78 | 0.40 |
+----------+------+------+------+

timm commented 8 years ago

This is kind of how NOT to present results.

dump 0.39. report 39.

compute variance pd, pf, g for TCA, bellwether

for each data sets, write a table with 1+3+3+3 columns

column 1 = name
column 2,3,4 = results from TCA+
column 5,6,7 = results from bellwether
column 8 = delta pd = 5 - 2
column 9 = delta pf = 6 - 3
column 10 = delta g = 7 - 4

note that for columns 8,9,10 if the delta is a "small effect" then print blank space, otherwise print the delta with a "+" or "-" sign in front

computing small effect

in the following, i,j are objects with knowledge of (say) pd for TCA+ and bellwether. they can report n (size) mu (mean) and sd (standard deviation)

def hedges(i,j,small=0.38):
    """
   Hedges effect size test. 
    Returns true if i,j differ by just a small effect. 
    I.e. there different is so small as to be dull!
    """
    num   = (i.n - 1)*i.s**2 + (j.n - 1)*j.s**2
    denom = (i.n - 1) + (j.n - 1)
    sp    = sqrt( num / denom )
    delta = abs(i.mu - j.mu) / sp  
    c     = 1 - 3.0 / (4*(i.n + j.n - 2) - 1)
    return delta * c < small

for justification of the above code, see equations 2,3,4 of and table 9 from https://pdfs.semanticscholar.org/e6be/263f60ccfb294e14422f0e0162b1367063a2.pdf

no, its not the best effect size test but it is one justified by an extensive literature review.

timm commented 8 years ago

Note that if you just as good as TCA+ but simpler than TCA+ is an over-elaboration

rahlk commented 8 years ago

Got it. I'll update the results accordingly.

rahlk commented 8 years ago

@timm

TCA vs. Bellwether: PDs, PFs, and Gs.

The Bellwether Method offers significantly lower False Alarm Rates. (Highlighted in Blue)
TCA+ has better PD scores. (Highlighted in Yellow)
Overall, when looking at G scores (Harmonic Mean of PD, PF), we see that Bellwether method is almost always better than TCA+. (Highlighted in Green)
The results are tabulated below. Here's a pdf version.

rahlk / Bellwether

TCA+ vs Bellwether #8

Summary (Now compiling....)

Comparisons

Target Project: ANT

TCA+

Bellwether Method

Target Project: LUCENE

TCA+

Bellwether Method

Target Project: JEDIT

TCA+

Bellwether Method

Target Project: XERCES

TCA+

Bellwether Method

Target Project: XALAN

TCA+

Bellwether Method

Target Project: CAMEL

TCA+

Bellwether Method

Target Project: VELOCITY

TCA+

Bellwether Method

Target Project: POI

TCA+

Bellwether Method

Target Project: LOG4J

TCA+

Bellwether Method

Target Project: IVY

TCA+

Bellwether Method

computing small effect

TCA vs. Bellwether: PDs, PFs, and Gs.

Results