Open itlittlekou opened 1 year ago
Hi, itlittlekou. You are right! We used news headlines in our experiment for time and cost reasons. Since news content contains most of the information, the best way is to use both the news headlines and news content.
However, in our experiment, we need to concat all the news related to certain stocks for a certain time period like one day. So the tokens might be super large and it might be quite difficult to train. We believe the best way might be to use the title and summary or just the title or summary might also work. If you have better plans, please don't hesitate to contact us or create PR!
How did you create the dataset? I don't see an api to do that.
I've noticed that while creating the dataset, the news headlines and news content were separated. This means that there are distinct training and testing sets for news headlines, as well as for news content. However, during the fine-tuning process, only the dataset containing news headlines was utilized, and the dataset with news content wasn't employed. Consequently, I'm somewhat perplexed about the role of the news content dataset in the fine-tuning process.
![image](https://github.com/AI4Finance-Foundation/FinGPT/assets/127939197/3612966e-0a50-4eda-82fe-26d024f0662a)