current12 / Stat-222-Project

2 stars 0 forks source link

Run McNemar's test on close accuracy results #94

Closed ijyliu closed 2 months ago

ijyliu commented 2 months ago

McNemar's test can allow us to tell if any close differences in accuracy are actually statistically significant. It's kind of similar to a chi-squared test and a paired t-test.

https://en.wikipedia.org/wiki/McNemar%27s_test

Suppose we want to check for a difference in accuracy between our Model A and our Model B. We essentially fill the contigency table with counts of test data calls correct by both model A and B, correct by model A but not B, correct by model B but not A, and incorrect by both models.

ijyliu commented 2 months ago

done, and can update as needed for more tests