search

OpenSourceEconomics / skillmodels

MIT License
11 stars 5 forks source link

Hit the 2000 columns limit in SQLite - stop logging fixed parameters #52

Closed hmgaudecker closed 3 years ago

hmgaudecker commented 4 years ago

In a kitchen-sink version of our serious model, we are trying to fit a model with 1.2k free parameters. An additional 1.4k parameters are fixed.

Maybe this is crazy, but a useful robustness check.

This currently fails with an OperationalError from SQLAlchemy because SQLite has a 2k columns limit.

Is it really necessary to log the fixed parameters or is this more for convenience?

[not sure whether this should go here or over to https://github.com/OpenSourceEconomics/estimagic]

janosg commented 4 years ago

We had a discussion whether it is better to log the internal or external parameter vector and decided for the external one for convenience. Maybe we should log a third type of parameter vector: those external parameters that are not fixed, explicitly or implicitly by equality constraints. All relevant information is in the processed params, so it would be a quick change.

However, I strongly dislike any kind of hard limits. You might go even crazier in the future and have more than 2000 free parameters. After quick googling I found that the limit with the pragma SQLITE_MAX_COLUMNS. I once played around with pragmas in sqlalchemy. It is not very difficult. The right place should be here

I will open a corresponding issue in estimagic.

janosg commented 3 years ago

Solved.