expectedparrot / edsl

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.
https://docs.expectedparrot.com
MIT License
168 stars 18 forks source link

Question `comment` is causing an error when applying survey memory rules #761

Closed rbyh closed 1 month ago

rbyh commented 2 months ago

Code to reproduce:

q_educ = QuestionMultipleChoice(
    question_name = "educ",
    question_text = "What is the highest level of education you have completed?",
    question_options = [
        "Did not graduate from high school",
        "High school graduate",
        "Some college, but no degree (yet)",
        "2-year college degree",
        "4-year college degree",
        "Postgraduate degree (MA, MBA, MD, JD, PhD, etc.)"
    ]
)

q_institution = QuestionFreeText(
    question_name = "institution",
    question_text = "Where did you complete your highest degree?"
)

survey = Survey([q_educ, q_institution]).add_targeted_memory(q_institution, q_educ)

results = survey.run()

Error:


Exception: InterviewErrorPriorTaskCanceled('Required tasks failed for institution')
Time: 1721058884.788631
Traceback:
Traceback (most recent call last):
  File "/Users/a16174/edsl/edsl/jobs/tasks/QuestionTaskCreator.py", line 230, in _run_task_async
    return await self._run_focal_task(debug)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/a16174/edsl/edsl/jobs/tasks/QuestionTaskCreator.py", line 125, in _run_focal_task
    requested_tokens = self.estimated_tokens()
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/a16174/edsl/edsl/jobs/tasks/QuestionTaskCreator.py", line 100, in estimated_tokens
    return self.token_estimator(self.question)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/a16174/edsl/edsl/jobs/interviews/InterviewTaskBuildingMixin.py", line 140, in _get_estimated_request_tokens
    for prompt in invigilator.get_prompts().values():
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/a16174/edsl/edsl/agents/PromptConstructionMixin.py", line 347, in get_prompts
    question_instructions=self.question_instructions_prompt,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/a16174/edsl/edsl/agents/PromptConstructionMixin.py", line 271, in question_instructions_prompt
    d[question].answer = answer
    ~^^^^^^^^^^
KeyError: 'educ_comment'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/a16174/edsl/edsl/jobs/interviews/Interview.py", line 139, in _extract_valid_results
    result = task.result()
             ^^^^^^^^^^^^^
  File "/Users/a16174/.pyenv/versions/3.11.7/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Users/a16174/.pyenv/versions/3.11.7/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/a16174/edsl/edsl/jobs/tasks/QuestionTaskCreator.py", line 239, in _run_task_async
    raise InterviewErrorPriorTaskCanceled(
edsl.exceptions.jobs.InterviewErrorPriorTaskCanceled: Required tasks failed for institution
johnjosephhorton commented 2 months ago

@rbyh Thank you - good catch. On it. In the interest of testing, in future, can you create code samples on chick? For example, a failing notebook object would be great.

rbyh commented 2 months ago

also happening for skip/stop rules, etc

rbyh commented 1 month ago

@johnjosephhorton https://chick.expectedparrot.com/content/c3835649-8d85-41b0-a218-41270a19d1ed