Closed epogrebnyak closed 8 years ago
Blocks in question
Writing parsed data to database:
Querying (somewhat convoluted):
@gabrielelanaro, @baor, @Pastafarianist , @alexanderlukanin13: need quick opinion
@epogrebnyak
freq
column.DateTimeIndex
- I don't see anything inherently evil in using Python function get_end_of_quarterdate
and get_end_of_monthdate
, as long as it doesn't noticeably affect performanceWhere labelled rows should be separated - in several database tables or (as done now) on querying of a single simple database table?
Because all labels have similar structure and in the code all labels are combined into one dataframe, storing them in one single table is the best way.
Where year, quarter and month should be converted to date?
I would recommend to move all "reshapes" to kep.database.db-module. kep.database.db should represent an adapter for database. Input for this module is link to database, output- filled pandas dataframe.
I side with @alexanderlukanin13. Annual/quarterly/monthly data is never combined, so it makes sense to get rid of freq
and also of NULL
s in SQLite tables. I think that's the simplest database design possible, which doesn't involve any hacks and thus it's most predictable and easy to maintain.
The script now imports flat labelled rows to database and separates annual, atr and monthly data on import (add link). Queries are run more often than import, maybe it should be other way around: write labelled rows to three tables, and query each table by get_dfa(), get_dfq(), get_dfm.
Design questions: