expectedparrot / edsl

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.
https://docs.expectedparrot.com
MIT License
182 stars 19 forks source link

Error when question name is also a word in the condition string for stop rule #769

Open rbyh opened 3 months ago

rbyh commented 3 months ago

Coop notebook: https://chick.expectedparrot.com/content/4f1a30c2-bbcf-4f33-96a6-9475333d5253

Code to reproduce ("union" is question name and also used in the text of the answer option referenced in the conditional logic):

from edsl import QuestionMultipleChoice, Survey

# From the CES pre-election questionnaire
q_union = QuestionMultipleChoice(
    question_name="**union**",
    question_text="Are you a member of a labor union?",
    question_options=[
        "Yes, I am currently a member of a labor union",
        "I formerly was a member of a labor union",
        "I am not now, nor have I been, a member of a labor union"
    ]
)

q_union_coverage = QuestionMultipleChoice(
    question_name="union_coverage",
    question_text="Are you covered by a union contract, also known as a collective bargaining agreement?",
    question_options=["Yes", "No", "Not sure"]
)

q_unionhh = QuestionMultipleChoice(
    question_name="unionhh",
    question_text="Other than yourself, is any member of your household a union member?",
    question_options=[
        "Yes, a member of my household is currently a union member",
        "A member of my household was formerly a member of a labor union, but is not now",
        "No, no one in my household has ever been a member of a labor union",
        "Not sure"
    ]
)

survey = (Survey([q_union, q_union_coverage, q_unionhh])
          .add_stop_rule(q_union, "**union** != 'Yes, I am currently a member of a labor **union**'")
         )

results = survey.run()

results.select("union", "union_coverage", "unionhh").print(format="rich")
rbyh commented 3 months ago

@johnjosephhorton What is the workaround? Can we auto-substitute it?

rbyh commented 3 months ago
Attempt 1 failed with exception:Exception in evaluation: invalid syntax (<unknown>, line 1). The expression is: union != 'Yes, I am currently a member of a labor union'. The current info env trying to substitute in is: {'union': 'I am not now, nor have I been, a member of a labor union', 'union_comment': 'I am not now, nor have I been, a member of a labor union'}. After the substition, the expression was: 'I am not now, nor have I been, a member of a labor union' != 'Yes, I am currently a member of a labor 'I am not now, nor have I been, a member of a labor union''. now waiting 1.00 seconds before retrying.Parameters: start=1.0, max=60.0, max_attempts=5.
johnjosephhorton commented 3 months ago

yeah - this is tough. One option would be to require references to question answers be in braces i.e.,

 .add_stop_rule(q_union, "{{ union }} != 'Yes, I am currently a member of a labor union'")

I kind of like this - thoughts?

rbyh commented 3 months ago

Yeah, I actually like that, because referencing the question name without quotes feels weird to me (ie "union == '....'" as opposed to "'union' == '....'")

johnjosephhorton commented 3 months ago

So this works as-is:

          .add_stop_rule(q_union, "{{ union }} != 'Yes, I am currently a member of a labor union'")
         )

I could do a warning whenever someone doesn't use braces for variables. Another thought - I could also insist that this be

"{{ union.answer }}

to make it very explicit. Thoughts?

rbyh commented 3 months ago

I like requiring union.answer for consistency with piping.