microsoft / rag-experiment-accelerator

The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.
https://github.com/microsoft/rag-experiment-accelerator
Other
187 stars 62 forks source link

02_qa_generation fails with exception #485

Open seekerofsai opened 6 months ago

seekerofsai commented 6 months ago

Running 02_qa_generation step fails with exception:

2024-04-15 20:42:21,690 - ERROR - rag_experiment_accelerator.ingest_data.acs_ingest - could not generate a valid json so moving over to next question! Error message: Expecting value: line 1 column 1 (char 0) 2024-04-15 20:42:21,720 - ERROR - rag_experiment_accelerator.ingest_data.acs_ingest - Traceback (most recent call last): File "/Users/<>/repos/rag-experiment-accelerator/rag_experiment_accelerator/ingest_data/acs_ingest.py", line 128, in generate_qna response_dict = json.loads( ^^^^^^^^^^^ File "/Users/<>/anaconda3/envs/rag-experiment/lib/python3.11/json/init.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/<>/anaconda3/envs/rag-experiment/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/<>/anaconda3/envs/rag-experiment/lib/python3.11/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

the generated response fails to load as json in generate_qna

'json[ { "question": "What are the top features and benefits of Surface Pro 8?", "answer": "Surface Pro 8s top features and benefits include being the most powerful pro ever, extending the ultimate desktop experience with Thunderbolt™ 4 ports, and having the most advanced display in a pro with improved video call capabilities." }, { "question": "How does Surface Pro 8s display enhance the user experience?", "answer": "Surface Pro 8s display is 11% larger, 10.8% higher resolution, 12.5% brighter, individually calibrated, virtually edge-to-edge, and now with up to 120Hz refresh rate for an even smoother pen experience and more responsive touch." }]'

denaha commented 6 months ago

It fails for me, with another error. The resources are configured properly, not sure which URL is called for "Completions", as the step is when it's calling the Completions URL. Any hints?

 Error message: RetryError[<Future at 0x37d707f50 state=finished raised NotFoundError>]
2024-04-18 06:11:05,680 - ERROR - rag_experiment_accelerator.ingest_data.acs_ingest - Traceback (most recent call last):
  File "/Users/deneha/Documents/github/azure/rag-experiment-accelerator/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/deneha/Documents/github/azure/rag-experiment-accelerator/rag_experiment_accelerator/llm/response_generator.py", line 71, in _create_chat_completion_with_retry
    return self.client.chat.completions.create(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deneha/Documents/github/azure/rag-experiment-accelerator/.venv/lib/python3.12/site-packages/openai/_utils/_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deneha/Documents/github/azure/rag-experiment-accelerator/.venv/lib/python3.12/site-packages/openai/resources/chat/completions.py", line 667, in create
    return self._post(
           ^^^^^^^^^^^
  File "/Users/deneha/Documents/github/azure/rag-experiment-accelerator/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1208, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deneha/Documents/github/azure/rag-experiment-accelerator/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 897, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/Users/deneha/Documents/github/azure/rag-experiment-accelerator/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 988, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}