Chinese blog about this project: 量化系列2 - 众包数据集
Table of contents generated with markdown-toc
To: dmnsn7 Who provided tushare token and make daily update possible.
wget https://github.com/chenditc/investment_data/releases/download/2023-10-08/qlib_bin.tar.gz
tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1
If you want to contribute to the set of scripts or the data, here is what you should do to set up a dev environment.
Follow https://github.com/dolthub/dolt
Raw data hosted on dolt: https://www.dolthub.com/repositories/chenditc/investment_data
To download as dolt database:
dolt clone chenditc/investment_data
docker run
-v /<some output directory>:/output
-it --rm chenditc/investment_data bash dump_qlib_bin.sh && cp ./qlib_bin.tar.gz /output/
You can use the following parameter to mount an existing dolt chenditc/investment_data folder to the container.
-v /<dolt directory>:/dolt
You will need tushare token to use tushare api. Get tushare token from https://tushare.pro/
export TUSHARE=<Token>
bash daily_update.sh
docker run -v /<some output directory>:/output -it --rm chenditc/investment_data bash daily_update.sh && bash dump_qlib_bin.sh && cp ./qlib_bin.tar.gz /output/
tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1
The database table on dolthub is named with prefix of data source, for example ts_a_stock_eod_price
. The meaning of the prefix:
The initial date for each stock might be different, when we calculate the adjusted price, we are using the first date price as adjust factor = 1.0.
In order to merge different data sources, we need to rescale the adjust factor, so that each data source will have the same adjusted price.
Each data source will have a dedicated link table, which is generated by:
Data validation needs to run to verify if the adjust factor match between each data source:
To add a new stock index, we need to change:
Please raise an issue to discuss the plan, example issue: https://github.com/chenditc/investment_data/issues/11
It should includes:
If the data is not clean, we might try hard to dig insight from it and find incorrect insight. So we want high quality data instead of just data.