Open christophscheuch opened 2 months ago
Can we still view the SQLite version after this migrate? It's very convenient to use SQLite database with python, and I can use DBeaver link to SQLite to see those data. Strongly recommend keep the SQLite version! Thanks!
@DayleLee we have not decided yet how to approach this topic and will take your feedback into consideration.
I just leave it here for future discussions: we might alternatively use duckdb since it has reached version 1.0.0: https://duckdb.org/why_duckdb. It also works nicely with arrow / parquet: https://r4ds.hadley.nz/arrow#using-duckdb-with-arrow. DuckDB can also be inspected with DBeaver.
Goal is to replace the SQLite database with parquet files that are shared between R and Python scripts, so that we have interoperability of data and remove redundancy.
To achieve this goal we need to do the following:
tidyfinance
package), e.g.: data/ ├── wrds/ │ ├── crsp_monthly.parquet │ ├── compustat.parquet ├── factors/ │ ├── ff5_monthly.parquet │ ├── q_daily.parquetarrow
andnanopartquet
pyarrow
andfastparquet