expectedparrot / edsl

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.
https://docs.expectedparrot.com
MIT License
97 stars 14 forks source link

Add 'piping' option where previous answers can be slotted into to question text / options in future questions #471

Open johnjosephhorton opened 1 month ago

johnjosephhorton commented 1 month ago
Screenshot 2024-05-11 at 6 22 34 AM
rbyh commented 1 month ago

Let's sketch out different piping situations:

  1. Simple: I want to (immediately) use the response to q1 as an input to q2 in the same survey.

  2. More complex: I want to extract themes/topics from each text in a set of texts (step 1), condense them (flatten list of lists) into a final non-duplicative substantively complete list (step 2), and then use that list as answer options to new questions (step 3).

--> Can this be done in a single survey?

  1. I want the model to generate a question that I then have another model answer. @benjamin-manning
benjamin-manning commented 1 month ago

3b. or have many other models answer

  1. I don't want the model to have a memory of entire previous questions, but I want to pipe in items that are answers from a subset of previous questions in a given survey into a new question as portions of that new questions---either in the question itself, or as answer options.

  2. Memory, but just a list of previous choices, no need for question repeats. Like in John's repeated prisoner's dilemma making it say "you responded to the following questions 6 times Question. Here are those responses: list of answers without repeating question

rbyh commented 1 month ago

I think this is the important distinction from targeted memory methods --- you want to access only the response, not the context of the question or fact that it was asked.

benjamin-manning commented 1 month ago

yup! exactly. I haven't used targeted memory much, but from my understanding, my request is different, and I hope, useful for people so worth implementing :)

rbyh commented 1 month ago

Current compose_questions() looks for the q1 question name to be present as a parameter of the q2 question text, but we should also allow the q1 answer to become the q2 answer options (multiple choice, checkbox, etc), ie modify this:

image
rbyh commented 1 month ago

eg using free text or list question to elicit a selection is not ideal because the agent is likely to differently format unless we add detailed formatting instructions:

image
rbyh commented 1 month ago

Note that we also do not want/need the field name duplicatively nested within the response to the composed/piped question:

image