QuestionFreeText generating JSON errors

[rbyh]

QuestionFreeText generating JSON errors in providing list responses (even without "list" specified in question_text) This is happening with GPT 3.5 and 4

[rbyh]

JSONDecodeError                           Traceback (most recent call last)
File ~/edsl_examples/venv/lib/python3.9/site-packages/edsl/language_models/LanguageModel.py:363, in LanguageModel.async_get_response(self, user_prompt, system_prompt)
    362 try:
--> 363     dict_response = json.loads(response)
    364 except json.JSONDecodeError as e:

File /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:

File /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
    333 """Return the Python representation of ``s`` (a ``str`` instance
    334 containing a JSON document).
    335 
    336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338 end = _w(s, end).end()

File /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py:353, in JSONDecoder.raw_decode(self, s, idx)
    352 try:
--> 353     obj, end = self.scan_once(s, idx)
    354 except StopIteration as err:

JSONDecodeError: Invalid control character at: line 1 column 348 (char 347)

[rbyh]

Same question_text for QuestionList has no errors:

[johnjosephhorton]

For the failing question, can you put a minimally reproducible example in a code block? E.g.,

from edsl import QuestionFreeText 
q = ...

I have a new procedure I'd like us to follow

[rbyh]

This problem also occurs with GPT 4 but here is minimal code that produced problem with default GPT 3.5:

from edsl.questions import QuestionFreeText

q = QuestionFreeText(
    question_name = "personas",
    question_text = "Draft detailed narratives for 5 personas likely to provide diverse responses to a public opinion poll about AI."
)

r = q.run()

[johnjosephhorton]

Please include model specification & imports so I can just copy & paste

[rbyh]

Done. To be clear, sometimes it does work - eg:

[johnjosephhorton]

that's fine - I'll probably write it in a loop w/o caching to try to force an exception

[johnjosephhorton]

your code references an m5?

[rbyh]

Didn't actually run that one commented out

[johnjosephhorton]

Great - so @apostolosfilippas @benjamin-manning - whenever we have a question where the JSON will not parse, please

1. Create an Issue and label as a bug
2. Put a minimally reproducible example in the issue, in a codeblock
3. If you're feeling up for it, add to edsl/integration/test_tricky_questions.py
4. Confirm failure

This will force us to solve problems here. Here's a first example: https://github.com/goemeritus/edsl/commit/037683d2de42531727570ec571d1f9355deb83cc