py-econometrics / pyfixest

Fast High-Dimensional Fixed Effects Regression in Python following fixest-syntax
https://py-econometrics.github.io/pyfixest/pyfixest.html
MIT License
118 stars 27 forks source link

Feols: speed up the creation of interacted fixed effects via `fe1^fe2` syntax #470

Closed s3alfisc closed 3 weeks ago

s3alfisc commented 1 month ago

Context

Currently, the creation of interacted fixed effects is extremely slow for large data sets.

    if "^" in fval:
        for val in fval.split("+"):
            if "^" in val:
                vars = val.split("^")
                data[val.replace("^", "_")] = data[vars].apply(
                    lambda x: (
                        "^".join(x.dropna().astype(str)) if x.notna().all() else np.nan
                    ),
                    axis=1,
                )

Likely, there is a way to speed this up =)

To Do

Re-implement _fixef_interactions and speed it up.

s3alfisc commented 3 weeks ago

@all-contributors please add @leostimpfle for code

s3alfisc commented 3 weeks ago

@all-contributors please add @leostimpfle for code

s3alfisc commented 3 weeks ago

Sorry, I had not activated the app for the py-econometrics project. One more attempt:

s3alfisc commented 3 weeks ago

@all-contributors please add @leostimpfle for code

allcontributors[bot] commented 3 weeks ago

@s3alfisc

I've put up a pull request to add @app! :tada:

I've put up a pull request to add @leostimpfle! :tada:

allcontributors[bot] commented 3 weeks ago

@s3alfisc

@leostimpfle already contributed before to code