search

forecastingresearch / forecastbench

A dynamic forecasting benchmark for LLMs
https://www.forecastbench.org
MIT License
5 stars 0 forks source link

Data: Yahoo Finance #3

Closed houtanb closed 5 months ago

houtanb commented 7 months ago
Source Tasks
Stocks US
Stocks non-US
YuehHanChen commented 7 months ago

I have code for getting stock data already, whoever works on this feel free to ask me for the code

houtanb commented 7 months ago

@ezrakarger also mentioned in slack:

and for stocks, there's an R package that links to public APIs so it should be a few lines, I think? I'm sure python has one too maybe to yahoo finance? google finance?

YuehHanChen commented 7 months ago

@ezrakarger also mentioned in slack:

and for stocks, there's an R package that links to public APIs so it should be a few lines, I think? I'm sure python has one too maybe to yahoo finance? google finance?

yeah my code is yahoo finance in python

zjac0bs commented 6 months ago

Example question:

Resolution criteria: Resolves to the market price at 11:59:59PM UTC on day of resolution.

houtanb commented 6 months ago

I would change the date in the question to be non American-centric. It would be parsed as April 2 by an American and Feb 4 by a European. It would be best to use the month name in the question instead of the month number for clarity.

houtanb commented 5 months ago

Will {ticker}'s market close price at resolution be higher than its market close price on the forecast date?

@zjac0bs @YuehHanChen Does the above phrasing sound good?

houtanb commented 5 months ago

Also, for the background field, what do you all think of: `"background": "S&P 500 market close prices as reported by Yahoo! Finance."

We could also imagine putting this background text there for the company if it's available via the API.

Screenshot 2024-05-13 at 17 26 54
YuehHanChen commented 5 months ago

Will {ticker}'s market close price at resolution be higher than its market close price on the forecast date?

@zjac0bs @YuehHanChen Does the above phrasing sound good?

soungs good to me

YuehHanChen commented 5 months ago

Also, for the background field, what do you all think of: `"background": "S&P 500 market close prices as reported by Yahoo! Finance."

We could also imagine putting this background text there for the company if it's available via the API. Screenshot 2024-05-13 at 17 26 54

S&P 500 market

Yes, yfinance has this company information. how about the following: "background" be the company background information returned by yfinance as suggested by @houtanb "value_at_freeze_datetime_explanation" be "The latest S&P 500 market close price of {ticker}."

So the background and the value_at_freeze_datetime_explanation are clearly different.

houtanb commented 5 months ago

Awesome! Glad they have that info. @YuehHanChen do you think it would be helpful for an LLM to have that info? @zjac0bs do you think it would be helpful for a human to have that info?

If not, then no need to include it.

If we don't include it, we could just put "{ticker} is/was a stock on the S&P 500. Market close prices are as reported by Yahoo! Finance." Use was if it has fallen off the S&P, is if it's still on there.

zjac0bs commented 5 months ago

@zjac0bs do you think it would be helpful for a human to have that info?

Definitely helpful for lesser known companies, though maybe less so for companies as well known as Apple? I would still argue for including - NVIDIA being as highly valued as it currently is would suggest that the correlation between public image and market value isn't 1:1. Plus I lean toward giving too much over too little!

YuehHanChen commented 5 months ago

agree with @zjac0bs For llm, it won't hurt

houtanb commented 5 months ago

Great! Go for it! Thanks.

So background could just be the company info and then "value_at_freeze_datetime_explanation": "The latest market close price of {ticker}." No need for S&P 500 here I think. If you want to say whether or not the stock is currently in the S&P 500, that can be included in the background section before the company overview. Otherwise, we can just drop that.