expectedparrot / edsl

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.
https://docs.expectedparrot.com
MIT License
184 stars 19 forks source link

Enable piping of individual items in a list `answer` into `question_options` #781

Closed rbyh closed 3 months ago

rbyh commented 3 months ago

Currently, items in an answer that is a list can be piped individually into a question_text but not question_options:

from edsl import QuestionList, QuestionFreeText, QuestionMultipleChoice, Survey

q1 = QuestionList(
    question_name = "q1",
    question_text = "What are your 3 favorite colors?",
    max_list_items = 3
)
q2 = QuestionFreeText(
    question_name = "q2",
    question_text = """Which color is your #1 favorite: 
    {{ q1.answer[0] }} or {{ q1.answer[1] }} or {{q1.answer[2] }}?"""
)
q3 = QuestionMultipleChoice(
    question_name = "q3",
    question_text = "Which color is your #1 favorite?",
    question_options = [
    "{{ q1.answer[0] }}",
    "{{ q1.answer[1] }}"
    ]
)

survey = Survey([q1,q2,q3])

results = survey.run()
results.select("q1","q2").print(format="rich")
Attempt 1 failed with exception:'q1' is undefined now waiting 1.00 seconds before retrying.Parameters: start=1.0, max=60.0, max_attempts=5.

Attempt 2 failed with exception:'q1' is undefined now waiting 2.00 seconds before retrying.Parameters: start=1.0, max=60.0, max_attempts=5.

Attempt 3 failed with exception:'q1' is undefined now waiting 4.00 seconds before retrying.Parameters: start=1.0, max=60.0, max_attempts=5.

Attempt 4 failed with exception:'q1' is undefined now waiting 8.00 seconds before retrying.Parameters: start=1.0, max=60.0, max_attempts=5.

Exceptions were raised in 1 out of 1 interviews.

---------------------------------------------------------------------------
UndefinedError                            Traceback (most recent call last)
Cell In[30], line 1
----> 1 results = survey.run()
      2 results.select("q1","q2").print(format="rich")

File [~/edsl/edsl/surveys/Survey.py:584](http://localhost:8888/lab/tree/~/edsl/edsl/surveys/Survey.py#line=583), in Survey.run(self, *args, **kwargs)
    567 """Turn the survey into a Job and runs it.
    568 
    569 Here we run a survey but with debug mode on (so LLM calls are not made)
   (...)
    580 └──────────────┘
    581 """
    582 from edsl.jobs.Jobs import Jobs
--> 584 return Jobs(survey=self).run(*args, **kwargs)

File [~/edsl/edsl/jobs/Jobs.py:502](http://localhost:8888/lab/tree/~/edsl/edsl/jobs/Jobs.py#line=501), in Jobs.run(self, n, debug, progress_bar, stop_on_exception, cache, check_api_keys, sidecar_model, batch_mode, verbose, print_exceptions, remote_cache_description, remote_inference_description)
    499     cache = Cache()
    501 if not remote_cache:
--> 502     results = self._run_local(
    503         n=n,
    504         debug=debug,
    505         progress_bar=progress_bar,
    506         cache=cache,
    507         stop_on_exception=stop_on_exception,
    508         sidecar_model=sidecar_model,
    509         print_exceptions=print_exceptions,
    510     )
    512     results.cache = cache.new_entries_cache()
    514     self._output(f"There are {len(cache.keys()):,} entries in the local cache.")

File [~/edsl/edsl/jobs/Jobs.py:586](http://localhost:8888/lab/tree/~/edsl/edsl/jobs/Jobs.py#line=585), in Jobs._run_local(self, *args, **kwargs)
    583 def _run_local(self, *args, **kwargs):
    584     """Run the job locally."""
--> 586     results = JobsRunnerAsyncio(self).run(*args, **kwargs)
    587     return results

File [~/edsl/edsl/utilities/decorators.py:62](http://localhost:8888/lab/tree/~/edsl/edsl/utilities/decorators.py#line=61), in jupyter_nb_handler.<locals>.wrapper(*args, **kwargs)
     59 if loop.is_running():
     60     # If the loop is running, schedule the coroutine and wait for the result
     61     future = asyncio.ensure_future(async_wrapper(*args, **kwargs))
---> 62     return loop.run_until_complete(future)
     63 else:
     64     # If the loop is not running, run the coroutine to completion
     65     return asyncio.run(async_wrapper(*args, **kwargs))

File [~/edsl/.venv/lib/python3.11/site-packages/nest_asyncio.py:98](http://localhost:8888/lab/tree/~/edsl/.venv/lib/python3.11/site-packages/nest_asyncio.py#line=97), in _patch_loop.<locals>.run_until_complete(self, future)
     95 if not f.done():
     96     raise RuntimeError(
     97         'Event loop stopped before Future completed.')
---> 98 return f.result()

File [~/.pyenv/versions/3.11.7/lib/python3.11/asyncio/futures.py:203](http://localhost:8888/lab/tree/~/.pyenv/versions/3.11.7/lib/python3.11/asyncio/futures.py#line=202), in Future.result(self)
    201 self.__log_traceback = False
    202 if self._exception is not None:
--> 203     raise self._exception.with_traceback(self._exception_tb)
    204 return self._result

File [~/.pyenv/versions/3.11.7/lib/python3.11/asyncio/tasks.py:277](http://localhost:8888/lab/tree/~/.pyenv/versions/3.11.7/lib/python3.11/asyncio/tasks.py#line=276), in Task.__step(***failed resolving arguments***)
    273 try:
    274     if exc is None:
    275         # We use the `send` method directly, because coroutines
    276         # don't have `__iter__` and `__next__` methods.
--> 277         result = coro.send(None)
    278     else:
    279         result = coro.throw(exc)

File [~/edsl/edsl/utilities/decorators.py:55](http://localhost:8888/lab/tree/~/edsl/edsl/utilities/decorators.py#line=54), in jupyter_nb_handler.<locals>.async_wrapper(*args, **kwargs)
     52 @functools.wraps(func)
     53 async def async_wrapper(*args, **kwargs):
     54     # This is an async wrapper to await the coroutine
---> 55     return await func(*args, **kwargs)

File [~/edsl/edsl/jobs/runners/JobsRunnerAsyncio.py:315](http://localhost:8888/lab/tree/~/edsl/edsl/jobs/runners/JobsRunnerAsyncio.py#line=314), in JobsRunnerAsyncio.run(self, cache, n, debug, stop_on_exception, progress_bar, sidecar_model, print_exceptions)
    313         shared_globals["edsl_runner_exceptions"] = task_history
    314         print(msg)
--> 315         task_history.html(cta="Open report to see details.")
    316         print(
    317             "Also see: https://docs.expectedparrot.com/en/latest/exceptions.html"
    318         )
    320 return results

File ~/edsl/edsl/jobs/tasks/TaskHistory.py:299, in TaskHistory.html(self, filename, return_link, css, cta)
    238 template = Template(
    239     """
    240 <!DOCTYPE html>
   (...)
    295 """
    296 )
    298 # Render the template with data
--> 299 output = template.render(
    300     interviews=self._interviews,
    301     css=css,
    302     performance_plot_html=performance_plot_html,
    303 )
    305 # Save the rendered output to a file
    306 with open("output.html", "w") as f:

File [~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py:1304](http://localhost:8888/lab/tree/~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py#line=1303), in Template.render(self, *args, **kwargs)
   1302     return self.environment.concat(self.root_render_func(ctx))  # type: ignore
   1303 except Exception:
-> 1304     self.environment.handle_exception()

File [~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py:939](http://localhost:8888/lab/tree/~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py#line=938), in Environment.handle_exception(self, source)
    934 """Exception handling helper.  This is used internally to either raise
    935 rewritten exceptions or return a rendered traceback for the template.
    936 """
    937 from .debug import rewrite_traceback_stack
--> 939 raise rewrite_traceback_stack(source=source)

File <template>:23, in top-level template code()

File [~/edsl/edsl/questions/QuestionBase.py:476](http://localhost:8888/lab/tree/~/edsl/edsl/questions/QuestionBase.py#line=475), in QuestionBase.html(self, scenario, include_question_name, height, width, iframe)
    468     question_content = Template("")
    470 base_template = Template(base_template)
    472 params = {
    473     "question_name": self.question_name,
    474     "question_text": Template(self.question_text).render(scenario),
    475     "question_type": self.question_type,
--> 476     "question_content": Template(question_content).render(scenario),
    477     "include_question_name": include_question_name,
    478 }
    479 rendered_html = base_template.render(**params)
    481 if iframe:

File [~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py:1304](http://localhost:8888/lab/tree/~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py#line=1303), in Template.render(self, *args, **kwargs)
   1302     return self.environment.concat(self.root_render_func(ctx))  # type: ignore
   1303 except Exception:
-> 1304     self.environment.handle_exception()

File [~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py:939](http://localhost:8888/lab/tree/~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py#line=938), in Environment.handle_exception(self, source)
    934 """Exception handling helper.  This is used internally to either raise
    935 rewritten exceptions or return a rendered traceback for the template.
    936 """
    937 from .debug import rewrite_traceback_stack
--> 939 raise rewrite_traceback_stack(source=source)

File <template>:4, in top-level template code()

File [~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py:487](http://localhost:8888/lab/tree/~/edsl/.venv/lib/python3.11/site-packages/jinja2/environment.py#line=486), in Environment.getattr(self, obj, attribute)
    483 """Get an item or attribute of an object but prefer the attribute.
    484 Unlike :meth:`getitem` the attribute *must* be a string.
    485 """
    486 try:
--> 487     return getattr(obj, attribute)
    488 except AttributeError:
    489     pass

UndefinedError: 'q1' is undefined
johnjosephhorton commented 3 months ago

Hint to self: options are only being rendered w/ the scenario, not piped answers.

File "/Users/john/tools/edsl/edsl/questions/QuestionMultipleChoice.py", line 54, in <listcomp>
    Template(str(option)).render(scenario) for option in self.question_options
johnjosephhorton commented 3 months ago

More hints - probably fix at this line in 'Invigilators'

answer = question._translate_answer_code_to_answer(response["answer"], scenario)
johnjosephhorton commented 3 months ago

At the breakpoint:

(Pdb) self.current_answers
{'q1': ['Blue', 'Green', 'Red'], 'q1_comment': 'These colors are widely appreciated for their depth and versatility.'}
(Pdb) self.question
Question('multiple_choice', question_name = """q3""", question_text = """Which color is your #1 favorite?""", question_options = ['{{ q1.answer[0] }}', '{{ q1.answer[1] }}'], model_instructions = {})
(Pdb) self.question.question_options
['{{ q1.answer[0] }}', '{{ q1.answer[1] }}']
(Pdb)