sergiocorreia
commented
1 year ago Hi Mario,
Thanks for spotting this. While I don't have a full answer yet, let me tell you what I know:
Despite foreign being a binary variable, there is a hdfe1 estimate for each of its two values as well as an estimate for the constant. How are the values determined?
With fixed effects, the constant is not materially relevant for reghdfe and ppmlhdfe, as the fixed effects made it unnecessary. However, people often asked for it, so what we did for reghdfe (and copied for ppmlhdfe) was to make it so the first set of fixed effects has a mean of zero, and assign its previous mean as a constant.
For instance:
sysuse auto
* Coefs are 8.711 and 8.761
glm price ibn.foreign, family(poisson) link(log) noconstant
* Replicate these numbers
ppmlhdfe price, a(FE=foreign)
replace FE = FE + _b[_cons]
tab FEHere, we can recover the same coefs as you would get with poisson or glm by adding back the constant.
Despite only one FE being used, hdfe1 and sumFE are not the same
You are correct, there seems to be a bit of numerical inaccuracy in sumFE, but I haven't been able to find any bug (yet).
sumFE seems to be different for every observation when comparing against a standard Poisson command, the predicted values are (slightly) different, using either estimate of the FE. Note that in this simple example ppmlhdfe does not drop any observation.
Yes, that's also what I get. If I had to speculate, I think the problem might be related to inaccuracies caused by data standardization. For instance, compare the two following commands:
* Default is to standardize data
sysuse auto, clear
ppmlhdfe price, a(FE=foreign) d(d) standardize_data(1)
tab FE d
sysuse auto, clear
ppmlhdfe price, a(FE=foreign) d(d) standardize_data(0)
tab FE dIn the first one, there six values of d, and in the second we correctly get only two values, which correspond to those of FE. We currently use a fast and not very accurate method for standardization (here as accuracy doesn't really matter here, but perhaps it does for ppmlhdfe (where there are lots of log and exp functions).
I'll keep researching on this, but in the mean time, depending on your code, it might work if you disable standardization as above.
Best, S
mariofiorini
Dear Sergio, Paulo and Thomas, I have posted this on Statalist, but GitHub might be the better place to so.
I am using your ppmlhdfe command (thanks for the time spent putting this together!) The goal is estimating a Poisson model with many levels of fixed effects (i.e. 4 categorical variables some of which are also interacted) that fails to converge using the conventional Poisson command, or even glm .. family(Poisson).
The ppmlhdfe command works well in the sense that i) it converges and ii) it is very fast. It does so by dropping singletons/separated observations. In the specifications where the Poisson command also converged, the point estimates are identical.
Next, I am trying to do some out of sample prediction, which the command does not allow for, so must be done manually by adding the estimated fixed effects. Here I am having some trouble understanding the output.
Example with only one binary FE and no other covariate: Code:
The options absorb(..., savefe) save all fixed effect estimates with __hdfe as prefix d(newvar) save sum of fixed effects as newvar; mandatory if running predict afterwards (except for predict,xb) Code:
So, I don’t understand why:
__hdfe1__estimate for each of its two values as well as an estimate for the constant. How are the values determined?__hdfe1__andsumFEare not the samesumFEseems to be different for every observationCode:
Any help would be great.