search

kingaa / pomp

R package for statistical inference using partially observed Markov processes
https://kingaa.github.io/pomp
GNU General Public License v3.0
110 stars 26 forks source link

feel confused when choosing the best fitted model with loglik #182

Closed stevenhuyi closed 1 year ago

stevenhuyi commented 1 year ago

I have some climatic covariates (e.g., temperature, humidity, and precipitation) and time seiries of hand, foot, and mouth disease (HFMD) data available. I intended to test if climatic factors drive the seasonality of HFMD. Hence, I used mif2 function to fit a SEIR model to the HFMD data. As I would like to test if lag effects of climatic factors on HFMD exist, zero week lag to four weeks lag were compared via loglik. The results are as follows:

loglik             loglik.se        lag
-1655.636495    0.770260243     0
-1646.292285    0.424285388     1
-1649.543875    3.008350385     2
-1644.713864    0.461131216     3
-1646.354714    0.346375666     4

According to the max logik, model of lag 3 is identified as the best fitting model. when simulating using paramers of lag 3, the result is poor as follows：

lag3

While the simulation of lag 0 is visually better as follows:

lag0

Hence, I'm really confused why this happened. Any reply will be much appreciated.

Yi

kingaa commented 1 year ago

Your question can be rephrased as "Why does the model that looks best to the likelihood not look best to me?" That is—assuming you have indeed maximized the likelihood—you are noticing that your "eyeball criterion" disagrees with the model's assessment.

Observe that your eyeball criterion weighs and summarizes the discrepancies between the data and the model. How does it do this? What function are you implicitly using to decide whether a given discrepancy is large or small?

One can also ask the same question of the model. Is it possible to plot the data and the model predictions in such a way that you can see what the model sees? Obversely, is it possible to formulate a likelihood function that matches your eyeball criterion? If you did so, what rationale would you give for that likelihood function?

Some suggestions:

  1. In envelope plots such as the ones you show above, the eye tends to reduce the model predictions to a uniform distribution, which is not what the model is actually saying. Moreover, you throw away all information about the model-predicted autocorrelation of trajectories. As I advised you before, it is wise to plot a small number (10–20) of representative trajectories, so that you can compare more of their features with those of the data. To get an idea of what the distribution looks like, consider also plotting a large number of trajectories as semitransparent traces, so that one can see where they are dense and where diffuse.
  2. It is usually possible to find a transformation of the data such that, on the scale of this transformation, the variance in the data (according to the model) is roughly independent of the mean. This is the scale on which vertical displacements of equal magnitude are equally likely. Therefore, this is the scale on which the model is looking at the data, i.e., the scale on which the eyeball criterion and the likelihood agree.