search

expectedparrot / edsl

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.
https://docs.expectedparrot.com
MIT License
97 stars 14 forks source link

Problems with non-Open AI models #462

Open rbyh opened 2 months ago

rbyh commented 2 months ago

I originally thought this was a problem with scenarios, but non-scenario questions are not running either:

from edsl.questions import QuestionMultipleChoice
from edsl import Model

model = Model('claude-3-opus-20240229')

q = QuestionMultipleChoice.example()

results = q.by(model).run()
image
rbyh commented 2 months ago
image
rbyh commented 2 months ago
image
rbyh commented 2 months ago 
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Question     ┃                                  ┃                   ┃                      ┃
┃ name         ┃ Exception                        ┃              Time ┃ Traceback            ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ how_feeling  │ Exception("Deep Infra response   │ 1715214725.213443 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.25301194190979004,             │                   │                      │
│              │ 'timestamp': 1715214725.213269,  │                   │                      │
│              │ 'cached_response': False,        │                   │                      │
│              │ 'cache_key':                     │                   │                      │
│              │ '2238a07cda21600f9dab7b19bab304… │                   │                      │
│ how_feeling  │ Exception("Deep Infra response   │  1715214726.22962 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.0021572113037109375,           │                   │                      │
│              │ 'timestamp': 1715214726.2294662, │                   │                      │
│              │ 'cached_response': True,         │                   │                      │
│              │ 'cache_key': None}")             │                   │                      │
│ how_feeling  │ Exception("Deep Infra response   │ 1715214728.247215 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.0029137134552001953,           │                   │                      │
│              │ 'timestamp': 1715214728.2469828, │                   │                      │
│              │ 'cached_response': True,         │                   │                      │
│              │ 'cache_key': None}")             │                   │                      │
│ how_feeling  │ Exception("Deep Infra response   │ 1715214732.265152 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.002839803695678711,            │                   │                      │
│              │ 'timestamp': 1715214732.264988,  │                   │                      │
│              │ 'cached_response': True,         │                   │                      │
│              │ 'cache_key': None}")             │                   │                      │
│ how_feeling  │ Exception("Deep Infra response   │ 1715214740.295725 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.009119033813476562,            │                   │                      │
│              │ 'timestamp': 1715214740.295447,  │                   │                      │
│              │ 'cached_response': True,         │                   │                      │
│              │ 'cache_key': None}")             │                   │                      │
│ how_feeling  │ InterviewErrorPriorTaskCanceled… │ 1715214740.297733 │                      │
│              │ tasks failed for how_feeling')   │                   │                      │
└──────────────┴──────────────────────────────────┴───────────────────┴──────────────────────┘