search

openai / evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Other
14.37k stars 2.54k forks source link

stock picking as model eval #1131

Open qrdlgit opened 1 year ago

qrdlgit commented 1 year ago

Describe the feature or improvement you're requesting

An idea for an eval which won't suffer from data contamination or overfitting.

maybe use financial news before the opening bell for some top S&P stocks + historical movements.

no need to trade, just use appropriate and standard portfolio metrics

at the very least, it would be an entertaining eval!

Additional context

No response

qrdlgit commented 1 year ago

Some more, brainstorming with gpt4:

Sports Betting or Fantasy Sports,Social Media Trend Prediction, Climate change temps, CDC stuff (flus, etc), Polling (rcp daily averages?)

An idea might be to use the various prediction markets around the web for sources of ideas and data.