search

forecastingresearch / forecastbench

A dynamic forecasting benchmark for LLMs
https://www.forecastbench.org
MIT License
9 stars 1 forks source link

Data: Serpapi #49

Open houtanb opened 6 months ago

houtanb commented 6 months ago

https://serpapi.com/

look into this, see if you can generate some good questions

YuehHanChen commented 6 months ago

Some ideas:

Technology Will [App] recieve more than [number] reviews on the App Store at each of the following resolution dates?

Entertainment Will [TV show] receive more than [number] votes on IMDb at each of the following resolution dates?

Social Media and Trends Will [influencer / Youtube Channel] gain more than [number] followers at each of the following resolution dates?

These can easily produce hundreds of questions.

Then every day, we use serpapi, to search key words, such as [App] reviews on the App Store [influencer / Youtube Channel] follower count [TV show] reviews on IMDb etc

The we prompt gpt 3.5 with the search result data returned from serpapi to determine the resolution every day. (if we do 300 questions per day, it'd cost ~$1.5 per day for using gpt 3.5)

YuehHanChen commented 6 months ago

For the cost from SerpAPI, it'd be $150 per month if we make around 300 calls per day https://serpapi.com/pricing

houtanb commented 6 months ago

Some ideas:

Technology Will [App] recieve more than [number] reviews on the App Store at each of the following resolution dates?

Entertainment Will [TV show] receive more than [number] votes on IMDb at each of the following resolution dates?

Social Media and Trends Will [influencer / Youtube Channel] gain more than [number] followers at each of the following resolution dates?

These can easily produce hundreds of questions.

Then every day, we use serpapi, to search key words, such as [App] reviews on the App Store [influencer / Youtube Channel] follower count [TV show] reviews on IMDb etc

The we prompt gpt 3.5 with the search result data returned from serpapi to determine the resolution every day. (if we do 300 questions per day, it'd cost ~$1.5 per day for using gpt 3.5)

I like the ideas here! I'd just be hesitant to use an LM to determine the resolution value when we want to be 100% sure about the resolution value. Why can't we get the resolution value directly from Serpapi?

Other ideas

How about something like google trends: https://serpapi.com/google-trends-interest-over-time

https://serpapi.com/google-trends-interest-by-region

https://serpapi.com/walmart-search-api

YuehHanChen commented 6 months ago

Some ideas: Technology Will [App] recieve more than [number] reviews on the App Store at each of the following resolution dates? Entertainment Will [TV show] receive more than [number] votes on IMDb at each of the following resolution dates? Social Media and Trends Will [influencer / Youtube Channel] gain more than [number] followers at each of the following resolution dates? These can easily produce hundreds of questions. Then every day, we use serpapi, to search key words, such as [App] reviews on the App Store [influencer / Youtube Channel] follower count [TV show] reviews on IMDb etc The we prompt gpt 3.5 with the search result data returned from serpapi to determine the resolution every day. (if we do 300 questions per day, it'd cost ~$1.5 per day for using gpt 3.5)

I like the ideas here! I'd just be hesitant to use an LM to determine the resolution value when we want to be 100% sure about the resolution value. Why can't we get the resolution value directly from Serpapi?

Other ideas

How about something like google trends: https://serpapi.com/google-trends-interest-over-time

  • Will interest in the term "X", as determined by Google, have increased by 10% by resolution date over it's current value?

https://serpapi.com/google-trends-interest-by-region

  • Will interest in the term "X" in region "Y", as determined by Google, have increased by 10% by resolution date over it's current value?

https://serpapi.com/walmart-search-api

  • What will the median price of all "X" products be at Walmart on the resolution date

Interesting, I didn't know they had this detailed info! I like these ideas more actually.