search

SJTU-DMTai / MASTER

This is the official code and supplementary materials for our AAAI-2024 paper: MASTER: Market-Guided Stock Transformer for Stock Price Forecasting. MASTER is a stock transformer for stock price forecasting, which models the momentary and cross-time stock correlation and guide feature selection with market information.
171 stars 36 forks source link

相同的模型在最新的qlib中IC得分比实验得分高出3倍是什么原因 #12

Closed LeonTing1010 closed 6 months ago

LeonTing1010 commented 6 months ago
Rank ICIR Rank IC ICIR IC step_len train_stop_loss_thred d_feat d_model t_nhead s_nhead n_epochs lr beta seed
0.8322383616317734 0.17739827966880373 0.891352351367328 0.1711398365432469 8 0.95 158 256 4 2 40 8e-06 10 0
zhiyuan5986 commented 6 months ago

Please check your version of qlib. We find that the TSDataSampler (in qlib.data.dataset.__init__) in v0.9.x is not correct, please use the TSDataSampler in v0.8.x instead. BTW, which version of code do you use (this repo or SJTUQUANT/qlib)?

LeonTing1010 commented 6 months ago

Please check your version of qlib. We find that the TSDataSampler (in qlib.data.dataset.__init__) in v0.9.x is not correct, please use the TSDataSampler in v0.8.x instead. BTW, which version of code do you use (this repo or SJTUQUANT/qlib)?

Yes, I am using SJTU-Quant, version 0.9.3.99.

zhiyuan5986 commented 6 months ago

So do you use config.sh to configure your environment? :)

LeonTing1010 commented 6 months ago

I installed it from the source code, which means I used pip install ..

zhiyuan5986 commented 6 months ago

You can see the README file in https://github.com/SJTU-Quant/qlib/tree/main/examples/benchmarks/MASTER, then try bash config.sh to configure your environment. We directly use our customizable qlib which fixes some bugs in original qlib.

LeonTing1010 commented 6 months ago

I only modified the TSDataSampler and obtained similar experimental results:

'IC': [0.06509559796801169, 0.059456687624642984], 'ICIR': [0.44744819247089423, 0.3800623345864442], 'Rank IC': [0.07775894785005492, 0.07454396389590388], 'Rank ICIR': [0.5411760191699715, 0.48359061178390356]

Do I still need to execute examples/benchmarks/MASTER/config.sh?

zhiyuan5986 commented 6 months ago

Yes. If you do not use config.sh, the system will directly use the qlib downloaded by pip.

LeonTing1010 commented 6 months ago

So can I directly use the Microsoft Qlib code, just restore TSDataSampler to version 0.8 to use the MASTER algorithm? Why do we need to copy the *.so files for the algorithm at https://github.com/SJTU-Quant/qlib/tree/main/examples/benchmarks/MASTER? It's Microsoft Qlib have other bugs that require changes to the .so files?

zhiyuan5986 commented 6 months ago

I only modified the TSDataSampler and obtained similar experimental results:

'IC': [0.06509559796801169, 0.059456687624642984], 'ICIR': [0.44744819247089423, 0.3800623345864442], 'Rank IC': [0.07775894785005492, 0.07454396389590388], 'Rank ICIR': [0.5411760191699715, 0.48359061178390356]

Do I still need to execute examples/benchmarks/MASTER/config.sh?

I'm sorry, this result really make sense, so your code is correct now :)

zhiyuan5986 commented 6 months ago

So can I directly use the Microsoft Qlib code, just restore TSDataSampler to version 0.8 to use the MASTER algorithm? Why do we need to copy the *.so files for the algorithm at https://github.com/SJTU-Quant/qlib/tree/main/examples/benchmarks/MASTER? It's Microsoft Qlib have other bugs that require changes to the .so files?

Yes, now you have no need to modify the code further. I'm sorry that I didn't see clearly for your second result.

LeonTing1010 commented 6 months ago

Thank you very much for your patient response. Your research findings have inspired me a lot.

zhiyuan5986 commented 6 months ago

You are welcome!