Readme

This is the official code and supplementary materials for our AAAI-2024 paper: MASTER: Market-Guided Stock Transformer for Stock Price Forecasting. [Paper] [ArXiv preprint]

MASTER is a stock transformer for stock price forecasting, which models the momentary and cross-time stock correlation and guides feature selection with market information.

Our original experiments were conducted in a complex business codebase developed based on Qlib. The original code is confidential and exhaustive. In order to enable anyone to quickly use MASTER and reproduce the paper's results, here we publish our well-processed data and core code.

Usage

1. Install dependencies.

   • pandas == 1.5.3
   • torch == 1.11.0

2. Install Qlib. We have minimized the reliance on Qlib, and you can simply install it by

   • pip install pyqlib
   • pylib == 0.9.1.99

3. Download data from one of the following links (the data files are the same) and unpack it into data/

   • OneDrive link
   • MEGA link
   • :fire:[New] Baidu link.

4. Run main.py.

5. We provide two trained models: model/csi300master_0.pkl, model/csi800master_0.pkl

Dataset

Form

The downloaded data is split into training, validation, and test sets, with two stock universes. Note the csi300 data is a subset of the csi800 data. You can use the following code to investigate the datetime, instrument, and feature formulation.

with open(f'data/csi300/csi300_dl_train.pkl', 'rb') as f:
    dl_train = pickle.load(f)
    dl_train.data # a Pandas dataframe

In our code, the data will be gathered chronically and then grouped by prediction dates. the data iterated by the data loader is of shape (N, T, F), where:

• N - number of stocks. For CSI300, N is around 300 on each prediction date; For CSI800, N is around 800 on each prediction date.
• T - length of lookback_window, T=8.
• F - 222 in total, including 158 factors, 63 market information, and 1 label.

Market information

For convenient reference, we extract and organize market information from the published data into data/csi_market_information.csv. You can check the datetime and feature formulation in the file. Note that m is shared by all stocks. The market data is generated by the following pseudo-code.

m = []
for S in csi300, csi500, csi800:
  m += [market_index(S,-1)]
  for d in [5, 10, 20, 30, 60]:
    m += [historical_market_index_mean(S, d), historical_market_index_std(S, d)]
    m += [historical_amount_mean(S, d), historical_amount_std(S, d)]

Preprocessing

The published data went through the following necessary preprocessing.

1. Drop NA features, and perform robust daily Z-score normalization on each feature dimension.
2. Drop NA labels and 5% of the most extreme labels, and perform daily Z-score normalization on labels.
   • Daily Z-score normalization is a common practice in Qlib to standardize the labels for stock price forecasting. To mitigate the difference between a normal distribution and groundtruth distribution, we filtered out 5\% of most extreme labels in training. Note that the reported RankIC compares the output ranking with the groundtruth, whose value is not affected by the label normalization.

:fire: [New] A Qlib implementation

We are happy to hear that MASTER has been integrated into the open-sourced Qlib framework at this repo. We thank LIU, Qiaoan and ZHAO, Lifan for their contributions and please also give credits to the new repo if you use it.

As a brief introduction to the new version, with the Qlib framework, you can

• report AR, IR, and more portfolio-based metrics,
• modify experiment configuration with .yaml files,
• compare with various models from the Qlib examples collection,
• benefit from other merits of Qlib.

In the meantime, please note that

• The new version utilizes a different data source published by Qlib, which covers a different timespan. The new data source is considered logically equal to our published data but may differ in values.
• The new version does not include the 'DropExtremeLabel' operation in data preprocessing but also reports decent performance.

Cite

If you use the data or the code, please cite our work! :smile:

@inproceedings{li2024master,
  title={Master: Market-guided stock transformer for stock price forecasting},
  author={Li, Tong and Liu, Zhaoyang and Shen, Yanyan and Wang, Xue and Chen, Haokun and Huang, Sen},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={1},
  pages={162--170},
  year={2024}
}