[BUG] error occurred while loading reghdfe.ado for version 6.12.1 27June2021 (but ssc version doesn't generate the error)

[flucoe]

Before submitting the bug report

1. Verify that you are using the latest versions of reghdfe and ftools (which reghdfe). Note that the latest version is usually on Github and not on SSC.
2. Verify that your Stata is updated (update query).

Bug report

Please complete the following information:

• Stata version: Stata 13.1
• OS: Linux

Behavior

• Expected behavior: install reghdfe from GitHub (following the instructions for manual installation) and that it works correctly. The version installed with ssc install (5.7.3 13nov2019 for reghdfe and 1.0.0 07jul2018 for ivreghdfe) works as expected.
• Actual behavior: what was the error. (this installs versions 6.12.1 27June2021 of reghdfe and 1.1.0 25Feb2021 of ivreghdfe using 4.1.11 22Nov2019 of ivreg2)

. cap ado uninstall reghdfe

. cap ado uninstall ivreghdfe

. 
. net install ftools, from(~/Downloads/ftools-master/src)
checking ftools consistency and verifying not already installed...
installing into /home/fernandoluco/ado/plus/...
installation complete.

. net install reghdfe, from(~/Downloads/reghdfe-master/src)
checking reghdfe consistency and verifying not already installed...
installing into /home/fernandoluco/ado/plus/...
installation complete.

. net install ivreghdfe, from(~/Downloads/ivreghdfe-master/src)
checking ivreghdfe consistency and verifying not already installed...
installing into /home/fernandoluco/ado/plus/...
installation complete.

. 
end of do-file

. sysuse auto, clear
(1978 Automobile Data)

. reghdfe price weight length, absorb(rep78)
invalid syntax
(error occurred while loading reghdfe.ado)
r(198);

end of do-file

r(198);

[dieterwang]

also occurs on version 6.12.2 02Nov2021

[sergiocorreia]

Hi Dieter, three questions:

1. Which version of Stata are you using? (you can type about; just remember to avoid pasting the last lines with your serial number)
2. Which version of ftools are you using? (which ftools)
3. If you type set trace on; set tracedepth 1, and then try running reghdfe again, can you see what line produces the error?

Best, S

[memoryfull]

I have Stata 13.1 and have encountered the same issue with reghdfe. My investigation (and previous issues) revealed that this is a combination of 3 problems:

• Use of adopath option in include calls that appeared in Stata >=14.
• Non-ASCII characters (Greek letters) in MATA procedures that were unsupported in Stata <14.
• Two parameters in fgetmatrix while Stata <=13 takes only one.

Having fixed these 3 issues I got the version 6 of reghdfe to work on Stata 13.1. Below is the walkthrough.

include, adopath

Having net installed the most recent version of reghdfe and its dependencies, I noticed that reghdfe and reghdfe5 issued the same cryptic error, and stack trace was unhelpful. However, regdhfe3 worked fine. This was a clear indication that some code regressions uncaught by unit tests occurred when version was bumped. Since version 6 of MATA procedures was a rewrite, it was not suprising.

I quickly caught the first problem, it was due to reghdfe.ado and reghdfe5.ado:

https://github.com/sergiocorreia/reghdfe/blob/3c5b73826fcdbbaa1a1a35227b91ad5887675ab8/src/reghdfe.ado#L844 https://github.com/sergiocorreia/reghdfe/blob/3c5b73826fcdbbaa1a1a35227b91ad5887675ab8/src/reghdfe5.ado#L528

Before Stata <14 include did not offer adopath option. The solution was to simply replace such calls with

cap findfile reghdfe5.mata
include "`r(fn)'"

cap findfile reghdfe.mata
include "`r(fn)'"

in the respective .ado files. To avoid expensive file lookups, one might reference the adopath directly:

local adodir : sysdir `"PLUS"'
include "`adodir'/r/reghdfe5.mata"

local adodir : sysdir `"PLUS"'
include "`adodir'/r/reghdfe.mata"

Greek letters

With this fix, I was able to run the include statement. Now MATA issued the following error:

>         // Type definitions (LSMR)
>         `Real'          α, β, ρ, θ, ζ, α_bar, ρ_bar, θ_bar, ζ_bar
'α' found where name expected
(263 lines skipped)
-----------------------------------------------
r(3000);

In Stata <14 MATA allowed only ASCII characters, and version 6 of reghdfe.mata uses the Greek letters thoughout the procedures, e.g.: https://github.com/sergiocorreia/reghdfe/blob/3c5b73826fcdbbaa1a1a35227b91ad5887675ab8/src/reghdfe.mata#L4493-L4494 or https://github.com/sergiocorreia/reghdfe/blob/3c5b73826fcdbbaa1a1a35227b91ad5887675ab8/src/reghdfe.mata#L4788-L4797

I replaced the following Greek letters with their ASCII counterparts in reghdfe.mata:

α -> alpha
β -> beta
γ -> gamma
δ -> delta
ρ -> rho
θ -> theta
φ -> phi
ζ -> zeta
τ -> tau
ψ -> psi

fgetmatrix

Having replaced the Greeks I re-ran the include statement

findfile reghdfe.mata
include "`r(fn)'"

and encountered the next error:

: // --------------------------------------------------------------------------
: // Load, partial out, and save data
: // --------------------------------------------------------------------------
: `Void' worker_partial_out(`String' hdfe_fn, `String' data_fn)
> {
>         `FixedEffects'                  HDFE
>         `Matrix'                                data
>         `Integer'               fh
> 
>         fh = fopen(hdfe_fn, "r")
>         HDFE = fgetmatrix(fh, 1)
wrong number of arguments for fgetmatrix()

It turns out that in Stata <14 the fgetmatrix accepted only one argument. The solution is to replace https://github.com/sergiocorreia/reghdfe/blob/3c5b73826fcdbbaa1a1a35227b91ad5887675ab8/src/reghdfe.mata#L5550 https://github.com/sergiocorreia/reghdfe/blob/3c5b73826fcdbbaa1a1a35227b91ad5887675ab8/src/reghdfe.mata#L5554 https://github.com/sergiocorreia/reghdfe/blob/3c5b73826fcdbbaa1a1a35227b91ad5887675ab8/src/reghdfe.mata#L5670

with fgetmatrix(fh).

Caveats

With these 3 fixes to reghdfe.ado and reghdfe.mata I managed to run the version 6 of reghdfe on Stata 13.1 without major errors. There are two caveats, though.

1. Factor expansion in interaction terms does not work:

sysuse auto
reghdfe price weight i.length, absorb(rep78) // works fine

reghdfe price weight##i.length, absorb(rep78) // produces an error
         _assert_abort():  3498  could not load data into Mata correctly; please contact author
            assert_msg():     -  function returned error
          st_data_pool():     -  function returned error
FixedEffects::partial_out():     -  function returned error
                 <istmt>:     -  function returned error

2. reghdfe5 still fails to execute. Even though I fixed the include call in reghdfe5.ado, I get the following after reghdfe5:

: `Void' BipartiteGraph::init(`FactorPointer' PF1,
>                             `FactorPointer' PF2,
>                             `Boolean' verbose)
> {
>         if (verbose) {
>                 printf("\n{txt}## Initializing bipartite graph\n\n")
>                 printf("    - FE #1: {res}%s{txt}\n", invtokens((*PF1).varlist))
type mismatch:  exp.exp:  transmorphic found where struct expected
(482 lines skipped)

This problem is harder to debug and I leave it for future investigation. I can make a PR with the suggested changes so that you can autotest them on newer versions (I am quite suspicious about my fgetmatrix fix, for instance).

I do understand that running reghdfe from 2022 on Stata 13.1 from 2013 is inherently problematic. Thank you for this amazing package and your time to address the community concerns.