Open jiffyclub opened 9 years ago
Low p-value is better right? Many of the p-values are very small, but 3 seem to be quite large. Might have to look at them individually and see what's going on. Keep in mind PUMS changed their sample this year, so we have a smaller sample than is typical - this could certainly make it harder to meet marginals. Would be interesting to look at marginals and joint distribution for those block groups that perform poorly.
Lower p-value is good when you're trying to prove two sets are not from the same distribution. In this case we're hoping the sets do look like the same distribution. For a goodness-of-fit test you want a low chi-squared. For example, the last geography above indicates a pretty good match between the synthetic totals and the target constraints. The first item is a poor match.
Ahh - well it looks like we have a problem then ;)
On Tue, Sep 16, 2014 at 4:37 PM, Matt Davis notifications@github.com wrote:
Lower p-value is good when you're trying to prove two sets are not from the same distribution. In this case we're hoping the sets do look like the same distribution. For a goodness-of-fit test you want a low chi-squared. For example, the last geography above indicates a pretty good match between the synthetic totals and the target constraints. The first item is a poor match.
— Reply to this email directly or view it on GitHub https://github.com/synthicity/synthpop/issues/22#issuecomment-55829857.
Also note that this is a "reduced" chi-squared, where I think values less than one are "pretty good" and values more than one are "not good".
I recorded some of the quality data from Napa County, which is pasted below. Low chi-squared is better (ideally less than 1) and high p-value is better. (Each indicating similarity between the expected and observed distributions.) One thing that stands out here is that some block groups turn out pretty well and others don't, and that that's repeatable between runs (it's not random chance). It seems like there's something about those particular block groups that help us end up with a good fit or poor fit that'll require some more investigation. I'm open for ideas on other ways of evaluating the final quality of the synthesis.