search

microsoft / qlib

Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.
https://qlib.readthedocs.io/en/latest/
MIT License
15.36k stars 2.63k forks source link

对于分布式训练的支持 #323

Open PKQ1688 opened 3 years ago

PKQ1688 commented 3 years ago

现阶段qlib是不是对分布式训练还不支持？

Derek-Wds commented 3 years ago

Hi @PKQ1688 , unfortunately Qlib doesn't support distributed training now, but we will try to add this feature in the future. Thanks~

you-n-g commented 3 years ago

@PKQ1688 The distributed training should be supported in the infrastructure level instead of the framework level. Qlib is a framework and doesn't restrict users from distributedly training models. You can implement your distributed version of model training in model.fit

PKQ1688 commented 3 years ago

好的 谢谢；我还想问一下 如果数据集过大，不能够一次读入内存 需要分批加载 数据 这种时候能够支持训练吗 还是需要自己改写数据处理部分。

you-n-g commented 3 years ago

@PKQ1688
Thanks for your feedbacks.

We don't have good support for this requirement. I aggree with you that it is a very important feature. I've included it into our plan.

PKQ1688 commented 3 years ago

好的 我先自己做一些改写，等我改写完了 给你们提pr。

PKQ1688 commented 3 years ago

Do you have any comments or suggestions for this work？

you-n-g commented 3 years ago

@PKQ1688 Looking forward to your PR! You can leverage some disk storage format which support seeking and slicing (e.g. numpy.array or PyTables).

It would be great if the new data layer could be transparent to other models.