FixedEffects / FixedEffectModels.jl

Fast Estimation of Linear Models with IV and High Dimensional Categorical Variables
Other
225 stars 46 forks source link

tolerance in `invsym!` #240

Closed w-zhiwei closed 1 year ago

w-zhiwei commented 1 year ago

The variable with very small variances will be dropped during the estimation, even though it's not collinear with others.

using DataFrames
using FixedEffectsModel

df = DataFrame(
    [36.9302  44.5105;
     39.4935  44.5044;
     38.946   44.5072;
     37.8005  44.5098;
     37.2613  44.5103;
     35.3885  44.5109;], 
     :auto
)

reg(df, @formula(x1 ~ x2))

For example, the above code will omit x2 but the coefficient can be correctly estmated in Stata.

                            Linear Model
=====================================================================
Number of obs:                  6   Degrees of freedom:             1
R2:                         0.000   R2 Adjusted:                0.000
F-Stat:                       NaN   p-value:                      NaN
=====================================================================
x1          | Estimate Std.Error t value Pr(>|t|) Lower 95% Upper 95%
---------------------------------------------------------------------
x2          |      0.0       NaN     NaN      NaN       NaN       NaN
(Intercept) |  37.6367  0.602309 62.4873    0.000   35.9644   39.3089
=====================================================================

The problem seems to be the torelance check in invsym!

https://github.com/FixedEffects/FixedEffectModels.jl/blob/851eca92998133fbb2780c4db1898c3f903d1d8f/src/utils/basecol.jl#L54-L74

Maybe I'm getting it wrong, but I'm not quite understand the check here. The diag value could be changed during the previous loops through line 66, but the tols variable is using the initial value.

matthieugomez commented 1 year ago

I now update the value for d in the presence of intercept, and with this modification, the program returns the same thing as Stata. https://github.com/FixedEffects/FixedEffectModels.jl/pull/241/files