Too many values to unpack (resp_text.split)

[lbux]

Describe the bug Sometimes, the evaluator returns null for score/explanation and sometimes it crashes with the following error:

2024-03-10 22:38:54.697 | INFO     | uptrain.framework.evalllm:evaluate:100 - Sending evaluation request for rows 0 to <50 to the Uptrain
2024-03-10 22:40:29.227 | INFO     | uptrain.framework.evalllm:evaluate:100 - Sending evaluation request for rows 0 to <50 to the Uptrain
2024-03-10 22:43:27.579 | ERROR    | uptrain.framework.remote:raise_or_return:32 - {"detail":"Error running the evaluation: 500: Error running the evaluation: concurrent.futures.process._RemoteTraceback: \n\"\"\"\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.10/concurrent/futures/process.py\", line 246, in _process_worker\n    r = call_item.fn(*call_item.args, **call_item.kwargs)\n  File \"/app/uptrain_server/worker.py\", line 425, in _run_log_and_eval\n    res = func(settings, dataset)\n  File \"/app/uptrain_server/evals/chatgpt.py\", line 383, in score_factual_accuracy\n    rationale, final_res = resp_text.split(\"Final result: \")\nValueError: too many values to unpack (expected 2)\n\"\"\"\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/app/uptrain_server/worker.py\", line 959, in run_eval_v2\n    result = future.result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 451, in result\n    return self.__get_result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 403, in __get_result\n    raise self._exception\nValueError: too many values to unpack (expected 2)\n"}

To Reproduce Steps to reproduce the behavior:

1. Use Factual Accuracy as a metric
2. Pass in several questions/context/responses
3. Provide valid OpenAI api key
4. Wait for results (for null values) or crash

Expected behavior There should be no null results with valid data and the evaluator should not crash when splitting.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context

I am using UpTrain through Haystack, but the issue seems to be on UpTrains side?

  File "C:\Users\lb\semantic_rag_haystack\evaluator.py", line 40, in <module>
    evaluation_results = evaluator_pipeline.run(evaluation_paramaters)
  File "C:\dev\anaconda3\envs\haystack\lib\site-packages\haystack\core\pipeline\pipeline.py", line 771, in run
    res = comp.run(**last_inputs[name])
  File "C:\dev\anaconda3\envs\haystack\lib\site-packages\haystack_integrations\components\evaluators\uptrain\evaluator.py", line 115, in run
    results = self._backend_client.evaluate(**eval_args)
  File "C:\dev\anaconda3\envs\haystack\lib\site-packages\uptrain\framework\evalllm.py", line 116, in evaluate
    raise e
  File "C:\dev\anaconda3\envs\haystack\lib\site-packages\uptrain\framework\evalllm.py", line 103, in evaluate
    batch_results = self.executor.evaluate(
  File "C:\dev\anaconda3\envs\haystack\lib\site-packages\uptrain\framework\remote.py", line 75, in evaluate
    raise e
  File "C:\dev\anaconda3\envs\haystack\lib\site-packages\uptrain\framework\remote.py", line 72, in evaluate
    response_json = raise_or_return(response)
  File "C:\dev\anaconda3\envs\haystack\lib\site-packages\uptrain\framework\remote.py", line 33, in raise_or_return
    response.raise_for_status()
  File "C:\dev\anaconda3\envs\haystack\lib\site-packages\httpx\_models.py", line 761, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'https://demo.uptrain.ai/api/open/evaluate_no_auth'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500

Thanks for submitting your bug – we love squashing them!

[Dominastorm]

Hey @lbux, thanks for bringing this up, we are looking into it!

[Abhiramkns]

Hello @Dominastorm, Can I work on this issue?

[Dominastorm]

Hey @Abhiramkns, I am working on this at the moment. Feel free to take up a different issue!