Open alexlafleur opened 7 years ago
There is no text here, just plots. Can this issue be closed?
These plots show the differences in errors from our error calculation (left) and the error calculation that was based on the count matrix from TransitionMatrixSampler(C=C1bayes). I think it can be closed.
There was something being tested in this check, and it doesn't look like we have brought this test to a satisfactory conclusion, unless that is documented elsewhere.
As I recall, the question was about how the error estimate that comes from the bayes approach - without taking advantage of our knowledge of the ground truth transition matrix - compares to the error estimate that does take advantage of that knowledge. In looking at these graphs, I recall that we found a bug in the test, because the behavior was not as expected. But I don't recall where the rest of the development was documented.
There is this script called "test_tmatrix_sampler", but we did not document anything for that as far as I remember.
I checked the script test_tmatrix_sampler. It is basically checking that the bayesian uncertainty estimation coming from the count matrix is roughly the same as the actual error that we calculate based on the known model. I think the plots are out of date in the issue, but in the end the test was succeeding - there is an assertion in there checking at the errors are not too different.(ratio between 1/4 and 4).