Open paulacanva opened 3 weeks ago
I'm seeing this error, too. it's unfortunate because we process thousands of records in a batch and if it fails for one call the whole thing goes down and we lose the data.
Hey @paulacanva there are a few anti patterns in your code. When you want to run many test cases on one metric, you should use the evaluate function: https://docs.confident-ai.com/docs/evaluation-introduction#evaluating-without-pytest
We have throttling options, and even ignoring errors so you don't lose your data on error like how @jyyogi described. You can also cache your results so even if it errors you don't have to rerun everything the next time.
The way you're using asyncio.gather will introduce a lot of bugs since each metric is stateful. In the evaluate function we handle everything so there's no overwriting of values.
hey @penguine-ip - thanks for the quick reply here! I'm using evaluate
and somehow glossed over the fact that we can cache our results as a way to get around failed calls. That feature is super nifty!
Do you have an example of how ignore_errors
+ write_cache
/ use_cache
behave?
This is extremely useful if you're running large amounts of test cases. For example, lets say you're running 1000 test cases using deepeval test run, but you encounter an error on the 999th test case. The cache functionality would allow you to skip all the previously evaluated 999 test cases, and just evaluate the remaining one.
How does this work? How would I retry?
❗BEFORE YOU BEGIN❗ Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt
Describe the bug No matter what I try, I keep getting
APIConnectionError: Connection error.
when using deepeval only.I have a script that first generates predictions, calling OpenAI, for ~400 rows. I use asyncio + semaphores + backoff to handle issues. Everything runs fine and fast.
When I get to the deepeval part, I'm able to process 3 of those rows before getting
I'm running the deepeval through:
I'm not sure why this issue just happens with deepeval and how to solve it.