search

yoavram / curveball

Predicting microbial growth in a mixed culture from growth curve data
http://curveball.yoavram.com
Other
14 stars 5 forks source link

Model fit benchmark #2

Closed yoavram closed 9 years ago

yoavram commented 9 years ago

Check that the best fit is better than a linear regression or log-linear regression in terms of BIC.

yoavram commented 9 years ago

I compare models fit to the fit to a linear model (OD(t) = y0 + a t). For one example (issue2.ipynb) I get:

Model BIC
Logistic5 -22.5613488612
Logistic6 -21.703153401
Logistic4 24.9089531996
Logistic3 28.8187761207
Lin2 29.3414487625

Some questions for @uriobolski to consider:

  1. Does BIC suppose to be >0?
  2. Can I say that BIC > 0? Means a bad fit?
  3. How much better does a BIC has to be, compared to the benchmark BIC (Lin2) to say that the fit is good?

Reminder: BIC = n log(RSS/n) + log(n) k where RSS is residual sum square, n is # of data points and k is number of model parameters.

uriobolski commented 9 years ago
  1. No.
  2. No: Basically, the BIC depends heavily on the number of data points you have, their variance, etc. It is by no means an absolute measure.
  3. Here is a table of values as a rule of thumb (for the gaussian case, which we indeed assume). Note that this does not mean that the fit is good, just that it is better (I can't really say significantly, but I can wink and mouth it silently).
yoavram commented 9 years ago

So based on the table you cited (which is taken from this paper we could set the benchmark BIC for a "good fit" as the Linear model BIC - 6. Or would you prefer -10? Both work pretty well for the example I made for this issue.

yoavram commented 9 years ago

BTW That BIC paper is cool. The first two applications they consider are about mutators in bacteria and hot hands in the NBA!! Perfect.

uriobolski commented 9 years ago

No, you can't say it's a good fit, but rather that the fitted curve has strong evidence that it is better than the benchmark if the delta BIC is larger than... The threshold is kind of flexible. The ecology guys use a minimum of delta AIC>2, so 6 in the BIC seems good enough, but it is really just a convention. What will happen in the software if the delta BIC is too small? Error? Warning? On Apr 30, 2015 9:25 AM, "Yoav Ram" notifications@github.com wrote:

So based on the table you cited (which is taken from this paper https://www.stat.washington.edu/raftery/Research/PDF/kass1995.pdf we could set the benchmark BIC for a "good fit" as the Linear model BIC - 6. Or would you prefer -10? Both work pretty well for the example I made for this issue.

— Reply to this email directly or view it on GitHub https://github.com/yoavram/curveball/issues/2#issuecomment-97677006.

yoavram commented 9 years ago

Right. We will use the benchmark to decide if the fit "succeeded". If it didn't we might want to try some heuristics to make it work or at report it failed.

uriobolski commented 9 years ago

Sounds good. On Apr 30, 2015 9:46 AM, "Yoav Ram" notifications@github.com wrote:

Right. We will use the benchmark to decide if the fit "succeeded". If it didn't we might want to try some heuristics to make it work or at report it failed.

— Reply to this email directly or view it on GitHub https://github.com/yoavram/curveball/issues/2#issuecomment-97680103.

yoavram commented 9 years ago

Possible heuristics when fit has failed is #9. Other possibilities are to fix y0 and or K.

yoavram commented 9 years ago

Implemented in commit ad986111413d31d429a8839e61110513cd60a065.