As Large Language Models (LLMs) become increasingly integrated into oureveryday lives, understanding their ability to comprehend human mental statesbecomes critical for ensuring effective interactions. However, despite therecent attempts to assess the Theory-of-Mind (ToM) reasoning capabilities ofLLMs, the degree to which these models can align with human ToM remains anuanced topic of exploration. This is primarily due to two distinct challenges:(1) the presence of inconsistent results from previous evaluations, and (2)concerns surrounding the validity of existing evaluation methodologies. Toaddress these challenges, we present a novel framework for procedurallygenerating evaluations with LLMs by populating causal templates. Using ourframework, we create a new social reasoning benchmark (BigToM) for LLMs whichconsists of 25 controls and 5,000 model-written evaluations. We find that humanparticipants rate the quality of our benchmark higher than previouscrowd-sourced evaluations and comparable to expert-written evaluations. UsingBigToM, we evaluate the social reasoning capabilities of a variety of LLMs andcompare model performances with human performance. Our results suggest thatGPT4 has ToM capabilities that mirror human inference patterns, though lessreliable, while other LLMs struggle.
URL
Affiliations
Abstract
Translation (by gpt-3.5-turbo)
Summary (by gpt-3.5-turbo)