amazon-science / RAGChecker

RAGChecker: A Fine-grained Framework For Diagnosing RAG
Apache License 2.0
530 stars 41 forks source link

RAGCHECKER ERROR : 'NoCredentialsError' object has no attribute 'response' #18

Closed FRAMEEE17 closed 1 month ago

FRAMEEE17 commented 1 month ago
    rag_results = []
    for input_item in checking_inputs['results']:
    query_id = input_item['query_id']
    query = input_item['query']
    gt_answer = input_item['gt_answer']

    try:
        response = solar.process_query(query, vector_db)

        rag_result = solar.response_to_eval(
            query_id=query_id,
            query=query,
            gt_answer=gt_answer,
            response=response
        )
        rag_results.append(rag_result)
    except Exception as e:
        print(f"Error processing query {query_id}: {str(e)}")
        print(f"Query: {query}")
        print(f"Response: {response}")

# Create RAGResults object
try:
    rag_results_obj = RAGResults.from_dict({"results": rag_results})
except Exception as e:
    print(f"Error creating RAGResults object: {str(e)}")
    print("rag_results:")
    for result in rag_results:
        print(json.dumps(result, indent=2))
    return
print(rag_results_obj)
# Initialize RAGChecker
evaluator = RAGChecker(
    extractor_name="bedrock/meta.llama3-70b-instruct-v1:0",
    checker_name="bedrock/meta.llama3-70b-instruct-v1:0",
    batch_size_extractor=32,
    batch_size_checker=32,
)

# Evaluate using RAGChecker
evaluator.evaluate(rag_results_obj, all_metrics)

```
DEBUG:LiteLLM:Standard Logging: created payload - payload: {'id': 'b1f38b4c-8fb1-415c-a02a-da77f6ea45e9', 'call_type': 
'completion', 'cache_hit': False, 'status': 'failure', 'saved_cache_cost': None, 'startTime': 1726979300.670663, 'endTime': 
1726979300.748564, 'completionStartTime': 1726979300.748564, 'model': 'meta.llama3-70b-instruct-v1:0', 'metadata': {}, 
'cache_key': None, 'response_cost': 0, 'total_tokens': 0, 'prompt_tokens': 0, 'completion_tokens': 0, 'request_tags': [], 'end_user': '', 
'api_base': None, 'model_group': '', 'model_id': '', 'requester_ip_address': None, 'messages': [{'role': 'user', 'content': 'Given a 
question and a candidate answer to the question, please extract a KG from the candidate answer condition on the question and 
represent the KG with triples formatted with ("subject", "predicate", "object"), each triplet in a line.\nPlease note that this is an 
EXTRACTION task, so DO NOT care about whether the content of the candidate answer is factual or not, just extract the triplets 
from it.\n\nHere are some in-context examples:\n\n### Question:\nGiven these paragraphs about the Tesla bot, what is its alias? 
\n\n### Candidate Answer:\nOptimus (or Tesla Bot) is a robotic humanoid under development by Tesla, Inc. It was announced at t 
he company\'s Artificial Intelligence (AI) Day event on August 19, 2021.\n\n### KG:\n("Optimus", "is", "robotic 
humanoid")\n("Optimus", "under development by", "Tesla, Inc.")\n("Optimus", "also known as", "Tesla Bot")\n("Tesla, Inc.", 
"announced", "Optimus")\n("Announcement of Optimus", "occurred at", "Artificial Intelligence (AI) Day event")\n("Artificial 
Intelligence (AI) Day event", "held on", "August 19, 2021")\n("Artificial Intelligence (AI) Day event", "organized by", "Tesla, 
Inc.")\n\n### Question:\nhere is some text about Andre Weiss, how many years was Andre at University of Dijon in Paris?\n\n### 
Candidate Answer:\n11 years\n\n### KG:\n("Andre Weiss at University of Dijon in Paris", "duration", "11 years")\n\n\nNow 
generate the KG for the following candidate answer based on the provided question:\n\n### Question:\nWhat is the Servicing Fee 
structure outlined in the CURO Receivables Finance II Servicing Agreement?\n\n### Candidate Answer:\nAccording to the CURO 
Receivables Finance II Servicing Agreement, the Servicing Fee is defined in Annex A. As stated in this section, the Servicing Fee for 
any Serviding to the CURO Receivables Finance II Servicing Agreement, the Servicing Fee is defined in Annex A. As stated in this 
section, the Servicing Fee for any Servicing Period is calculated as the product of (a) 2.00%, (b) the daily average Receivable 
Balance of all Receivables serviced under the agreement during the Servicing Period is calculated as the product of (a) 2.00%, (b) 
the daily average Receivable Balance of all Receivables serviced under the agreement during the Servicing Period, and (c) a fraction 
where the numerator is the number of calendar days during the Servicing Period and the denominator is 360.\n\n### KG:\n'}], 
'response': {}, 'model_parameters': {'temperature': 1e-05, 'maxTokens': 1000}, 'hidden_params': {'model_id': None, 'cache_key': None, 
'api_base': None, 'response_cost': None, 'additional_headers': None}, 'model_map_information': {'model_map_key': '', 
'model_map_value': None}, 'error_str': "'NoCredentialsError' object has no attribute 'response'"}
ing Period, and (c) a fraction where the numerator is the number of calendar days during the Servicing Period and the denominator 
is 360.\n\n### KG:\n'}], 'response': {}, 'model_parameters': {'temperature': 1e-05, 'maxTokens': 1000}, 'hidden_params': {'model_id': 
None, 'cache_key': None, 'api_base': None, 'response_cost': None, 'additional_headers': None}, 'model_map_information': 
{'model_map_key': '', 'model_map_value': None}, 'error_str': "'NoCredentialsError' object has no attribute 'response'"}
ponse': {}, 'model_parameters': {'temperature': 1e-05, 'maxTokens': 1000}, 'hidden_params': {'model_id': None, 'cache_key': None, 
'api_base': None, 'response_cost': None, 'additional_headers': None}, 'model_map_information': {'model_map_key': '', 
'model_map_value': None}, 'error_str': "'NoCredentialsError' object has no attribute 'response'"}

    Exit with the following error:
    'NoCredentialsError' object has no attribute 'response'
      0%|   

how to solve this issue and what does this error ??

FRAMEEE17 commented 1 month ago

is it because I couldn't access to extractor and checker model?

FRAMEEE17 commented 1 month ago
image

Now, I use GPT-4O mini and it solved the first error I think , and I get this error like this image.

FRAMEEE17 commented 1 month ago
image

Line 166