test addition - Githubissues

caporaso-lab / sourcetracker2

SourceTracker2

BSD 3-Clause "New" or "Revised" License

62 stars 45 forks source link

test addition #61

Closed wdwvt1 closed 8 years ago

wdwvt1 commented 8 years ago

@gregcaporaso and I pair reviewed and decided there are a couple keys places where test coverage should be improved.

[x] Addition of tests of the entire package against SourceTracker 1. We can do this simply by building several tables, running with ST1, and comparing the distribution of proportions to the distribution of proportions we'd get from ST2. This will be a stochastic test, and we will have to set our acceptable error rate (alpha=.05) where tests will fail but no significant change has taken place. Something like the following.

from scipy.stats import ttest_1samp
st1_ms = np.array([[.3, .4, .3],[.2, .3, .5]])
st2_ms = gibbs(source, sinks)

obs_t, obs_p = ttest_1samp(st2_ms, st1_ms.mean(0))
p_threshhold = .05
self.assertTrue(obs_p > p_threshhold)

[x] Addition of tests for gibbs and gibbs_loo (these are the largest sources of untested code in the entire package). We'll have to make a hand rolled example that we run through from start to finish.
[x] Consistency tests for gibbs and gibbs_loo - these will just test that when a code change happens, a PRNG seeded examples didn't change. These do not test accuracy, just consistency.

wdwvt1 commented 8 years ago

The pr #67 adds the consistency tests. Working on adding the hand rolled examples now.