ForecastBench

A forecasting benchmark for LLMs. Leaderboards and datasets available at https://www.forecastbench.org.

Getting started

Local setup

1. git clone --recurse-submodules <repo-url>.git
2. cd llm-benchmark
3. cp variables.example.mk variables.mk and set the values accordingly
4. Setup your Python virtual environment
   1. make setup-python-env
   2. source .venv/bin/activate

Run GCP Cloud Functions locally

1. cd directory/containing/cloud/function
2. eval $(cat path/to/variables.mk | xargs) python main.py

Contributions

Before creating a pull request:

• run make lint and fix any errors and warnings
• ensure code has been deployed to Google Cloud Platform and tested (only for our devs, for others, we're happy you're contributing and we'll test this on our end).
• fork the repo
• reference the issue number (if one exists) in the commit message
• push to the fork on a branch other than main
• create a pull request