search

ag-csw / LDStreamHMMLearn

1 stars 0 forks source link

TMatrixSampler Error Comparison #6

Open alexlafleur opened 7 years ago

alexlafleur commented 7 years ago

error_comparison

greenTara commented 7 years ago

There is no text here, just plots. Can this issue be closed?

alexlafleur commented 7 years ago

These plots show the differences in errors from our error calculation (left) and the error calculation that was based on the count matrix from TransitionMatrixSampler(C=C1bayes). I think it can be closed.

greenTara commented 7 years ago

There was something being tested in this check, and it doesn't look like we have brought this test to a satisfactory conclusion, unless that is documented elsewhere.

greenTara commented 7 years ago

As I recall, the question was about how the error estimate that comes from the bayes approach - without taking advantage of our knowledge of the ground truth transition matrix - compares to the error estimate that does take advantage of that knowledge. In looking at these graphs, I recall that we found a bug in the test, because the behavior was not as expected. But I don't recall where the rest of the development was documented.

alexlafleur commented 7 years ago

There is this script called "test_tmatrix_sampler", but we did not document anything for that as far as I remember.

greenTara commented 7 years ago

I checked the script test_tmatrix_sampler. It is basically checking that the bayesian uncertainty estimation coming from the count matrix is roughly the same as the actual error that we calculate based on the known model. I think the plots are out of date in the issue, but in the end the test was succeeding - there is an assertion in there checking at the errors are not too different.(ratio between 1/4 and 4).