sergiocorreia / reghdfe

Linear, IV and GMM Regressions With Any Number of Fixed Effects
http://scorreia.com/software/reghdfe/
MIT License
219 stars 57 forks source link

assert(): 3498 assertion is false #104

Open Danferno opened 7 years ago

Danferno commented 7 years ago

I'm not sure what has changed, because I ran this code earlier and it worked, but now I get an error with following syntax (there are five fstacked* and five lambda variables) ` reghdfe ln_E_rest L1_ln_E_rest L2_ln_E_rest L6_ln_E_rest ln_MW L1_ln_MW L2_ln_MW L6_ln_MW ln_pop ln_E_tot, absorb(County#c.f_stacked month#c.lambda_* County month)`

                assert():  3498  assertion is false
         fixed_effects():     -  function returned error
                 <istmt>:     -  function returned error

I also tried reghdfe ln_E_rest L1_ln_E_rest L2_ln_E_rest L6_ln_E_rest ln_MW L1_ln_MW L2_ln_MW L6_ln_MW ln_pop ln_E_tot, absorb(County##c.f_stacked* month##c.lambda_*)

But then the interactions are only kept for the first instance of each variable (output below explains this better than I can in words)

Absorbed degrees of freedom:
-----------------------------------------------------------------+
             Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------------------+---------------------------------------|
                  County |      1030           0        1030     |
     County#c.f_stacked1 |      1030           0        1030    ?|
              f_stacked2 |       282           1         281     |
              f_stacked3 |       282         282           0    ?|
              f_stacked4 |       282         282           0    ?|
              f_stacked5 |       282         282           0    ?|
                   month |       282         282           0    ?|
 month#c.lambda_stacked1 |       282           0         282    ?|
         lambda_stacked2 |      1030        1030           0    ?|
         lambda_stacked3 |      1030        1030           0    ?|
         lambda_stacked4 |      1030        1030           0    ?|
         lambda_stacked5 |      1030        1030           0    ?|
-----------------------------------------------------------------+

Is there something I'm doing wrong? I'm using Stata 14.2, reghdfe version 4.3.6 28jun2017

PS: This seems to be where things go wrong:


    - mata: HDFE = fixed_effects(`"`absorb' `comma' `options'"', "`touse'", "`weight'", "`exp'", `drop_singletons', `verbose')
    = mata: HDFE = fixed_effects(`"County#c.f_stacked* month#c.lambda_* County month , "', "__000001", "", "", 1, 0)
                assert():  3498  assertion is false
         fixed_effects():     -  function returned error
                 <istmt>:     -  function returned error
    ---------------------------------------------------------------------------------------------------------------------------------------------- end reghdfe.Parse ---
  --------------------------------------------------------------------------------------------------------------------------------------------- end reghdfe.Estimate ---
- Cleanup `c(rc)'
= Cleanup 3498

PS2: Manually expanding the * seems to work: reghdfe ln_E_rest L1_ln_E_rest L2_ln_E_rest L6_ln_E_rest ln_MW L1_ln_MW L2_ln_MW L6_ln_MW ln_pop ln_E_tot, absorb(${panelvar}##c.(f_stacked1 f_stacked2 f_stacked3 f_stacked4 f_stacked5) ${timevar}##c.(lambda_stacked1 lambda_stacked2 lambda_stacked3 lambda_stacked4 lambda_stacked5))


-----------------------------------------------------------------+
             Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------------------+---------------------------------------|
                  County |      1030           0        1030     |
     County#c.f_stacked1 |      1030           0        1030    ?|
     County#c.f_stacked2 |      1030           0        1030    ?|
     County#c.f_stacked3 |      1030           0        1030    ?|
     County#c.f_stacked4 |      1030           0        1030    ?|
     County#c.f_stacked5 |      1030           0        1030    ?|
                   month |       282           1         281     |
 month#c.lambda_stacked1 |       282           0         282    ?|
 month#c.lambda_stacked2 |       282           0         282    ?|
 month#c.lambda_stacked3 |       282           0         282    ?|
 month#c.lambda_stacked4 |       282           0         282    ?|
 month#c.lambda_stacked5 |       282           0         282    ?|
-----------------------------------------------------------------+
sergiocorreia commented 7 years ago

I don't recall if the code in absorb allows * expansions; at least I haven't used them before.

Also, I would strongly suggest if you replace "country#var country" with "country##var), it makes a large difference on speed and numerical accuracy.

sergiocorreia commented 7 years ago

Edit: I see you are already doing ## in the last example.

Danferno commented 7 years ago

Here's an older log (I don't know which reghdfe version I used back then)

. reghdfe $depvar ${Lx_$depvar} ${Lx_$indepvar} $controls, absorb(${panelvar}##c.f_stacked* ${timevar}##c.lambda_*) cluster($clustervar) residuals(res_`name')            // Clustered
(converged in 3 iterations)

Absorbed degrees of freedom:
----------------------------------------------------------------------------+
              Absorbed FE |  Num. Coefs.  =   Categories  -   Redundant     | 
--------------------------+-------------------------------------------------|
                   County |            0            1030           1030 *   | 
      County#c.f_stacked1 |         1030            1030              0 ?   | 
      County#c.f_stacked2 |         1030            1030              0 ?   | 
      County#c.f_stacked3 |         1030            1030              0 ?   | 
      County#c.f_stacked4 |         1030            1030              0 ?   | 
      County#c.f_stacked5 |         1030            1030              0 ?   | 
                    month |          281             282              1     | 
  month#c.lambda_stacked1 |          282             282              0 ?   | 
  month#c.lambda_stacked2 |          282             282              0 ?   | 
  month#c.lambda_stacked3 |          282             282              0 ?   | 
  month#c.lambda_stacked4 |          282             282              0 ?   | 
  month#c.lambda_stacked5 |          282             282              0 ?   | 
----------------------------------------------------------------------------+

For now I just fixed it by using the ds command, but I don't know how convenient that is in an ado/mata context ...

ds f_stacked*
local fvars = "`r(varlist)'"

ds lambda_*
local lambdavars = "`r(varlist)'"
reghdfe ln_E_rest L1_ln_E_rest L2_ln_E_rest L6_ln_E_rest ln_MW L1_ln_MW L2_ln_MW L6_ln_MW ln_pop ln_E_tot, absorb(${panelvar}##c.(`fvars') ${timevar}##c.(`lambdavars'))
sergiocorreia commented 7 years ago

By the way, the code that parses absorb() is actually in ftools and you can access it directly from Stata:

sysuse auto, clear
ms_parse_absvars foreign##c.*h*, noisily
ms_parse_absvars foreign##c.(*h*), noisily

It's in it's own .ado so it can be called from other programs (e.g. for IV regression), but that also makes it easier to fix.

In this case, with parenthesis it works correctly but without them it forgets to add a parenthesis; fixing it might be fairly straightforward.

Danferno commented 7 years ago

Ah, good to know.

Just FYI, this also causes issues with the regife command.

regife $depvar $indepvar $controls, f(lambdai = $panelvar ft = $timevar, 7) a(uPart = $panelvar tpart = $timevar) vce(cluster $clustervar) residuals(res_`name')

internall call to reghdfe failed (error code: 3498). Returning the estimate without standard errors.

REGIFE                                            Number of obs   =          .
Panel structure: County, month                    Wald chi2()     =          .
Factor dimension: 7                               Prob > chi2     =          .
Converged: true                                   Iterations      =       1681

last estimates not found
r(301);
sergiocorreia commented 7 years ago

TBH the new version causes issues with every command that depends on reghdfe (regife, poi2hdfe, etc.), which is one of the reasons I haven't updated the SSC version.

Danferno commented 7 years ago

Oh the joys of software development... :D

On 14 July 2017 at 03:33, Sergio Correia notifications@github.com wrote:

TBH the new version causes issues with every command that depends on reghdfe (regife, poi2hdfe, etc.), which is one of the reasons I haven't updated the SSC version.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sergiocorreia/reghdfe/issues/104#issuecomment-315245262, or mute the thread https://github.com/notifications/unsubscribe-auth/AUBCSjEJ8FuUaKu3Ou8-UOEkkZ_7T4TRks5sNsWAgaJpZM4OVb04 .

glennmagerman commented 7 years ago

Hi guys, FYI - I have the July 13, 2017 version installed, and it crashes Stata every time I use reghdfe - unless using the option "old". System: OSX El Capitan (both on iMac and on macbook pro). On the windows server it does not seem to cause problems.

sergiocorreia commented 7 years ago

Hi Glenn,

What happens if you type "reghdfe, compile"? (And then maybe close and open stata)

glennmagerman commented 7 years ago

Hi Sergio,

thanks, works after a reboot of the system! I must have missed that information, apologies.

sergiocorreia commented 7 years ago

Hi Glenn,

I haven't heard of that problem before, but restarting usually solves most problems :) (although I am worried that when I upload to SSC I could get many issue reports)

glennmagerman commented 7 years ago

I had to compile and restart the system in the end... Happy to send feedback. cheers Glenn