This is the code repository for the Arxiv paper SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluation
Use virtual environment tools (e.g miniconda) to install packages and run experiments\ python==3.7.10\ pip install -r requirements.txt
The code is organized as follows:
Change the /path/to/working/dir to the path to your working directory.
You need to export your own OpenAI API key before running experiments with OpenAI API, i.e., export OPENAI_API_KEY=$YOUR_OPENAI_API_KEY
Following the instructions in ROSCOE code repository:
Processing the data for our experiments:
python roscoe_data_processing.py
To reproduce the experiments for GPT-4 evaluation:
python gpt4_evaluation_gsm8k.py
python gpt4_evaluation_esnli.py
python gpt4_evaluation_drop.py
python gpt4_evaluation_cosmos.py
To reproduce the experiments for SocREval:
python SocREval_gsm8k.py
python SocREval_esnli.py
python SocREval_drop.py
python SocREval_cosmos.py