Enhancement: Can we make loading of the ANES dataset faster?

[josh-ashkinaze]

@narenedara It is not an urgent thing for this week but can you think about how to make ANES dataset processing faster? Just consider this situation:

from plurals.deliberation import Chain, Ensemble, Debate

issue = "Should the United States ban assault rifle words? Answer in 20 words."
model = 'gpt-4o'

n_per_group = 20

default_conservative = [Agent(persona="A conservative voter", model=model) for _ in range(n_per_group)]
default_liberal = [Agent(persona="A liberal voter", model=model) for _ in range(n_per_group)]

• [x] We should load ANES automatically when importing the module like how we load in DEFAULTS, make it available to all agents. From a user experience perspective, I think it is better to have slightly longer initial import time and then have each individual thing slower.

• [x] The way we implement ideology is by just using the query_str method. But I am pretty sure that searching with strings has some non-zero lag above slicing the dataframe because minimally Pandas needs to decode the string. It's not really noticeable with one agent but it is with many. So in a next release we probably want to not use query_str for ideology if we do not have to.

[josh-ashkinaze]

EDIT: I am working on this now and will close. My new method loads ANES at the start using GLOBALS and then I reverted back to slicing dataframe instead of string querying. it has passed our unit tests.