dosubot[bot]
commented
3 months ago Hello, @thistleknot! I'm here to help you with your bug. I'm an bot that can help you solve bugs, answer questions, and become a contributor. Let's get to the bottom of this together!
To resolve the openai.AuthenticationError with error code 401, ensure that the OPENAI_API_KEY environment variable is set correctly with a valid OpenAI API key. Here is the relevant code snippet to set the API key:
import os
from getpass import getpass
import openai
# Set your OpenAI API key if it is not already set as an environment variable
if not (openai_api_key := os.getenv("OPENAI_API_KEY")):
openai_api_key = getpass("🔑 Enter your OpenAI API key: ")
openai.api_key = openai_api_key
os.environ["OPENAI_API_KEY"] = openai_api_keyMake sure to replace "your-openai-key" with your actual OpenAI API key. If the key is incorrect or missing, you will encounter the openai.AuthenticationError.
Additionally, you can pass raise_exceptions=False in the evaluate function to show only a warning message instead of raising an exception:
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import context_recall
data_samples = {
'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'],
['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}
dataset = Dataset.from_dict(data_samples)
# Ensure the API key is correct
os.environ["OPENAI_API_KEY"] = "your-valid-openai-key"
score = evaluate(dataset, metrics=[context_recall], raise_exceptions=False)
score.to_pandas()This will prevent the ExceptionInRunner from stopping your execution and will only show a warning message [1][2].
To continue talking to Dosu, mention @dosu.
thistleknot
[ ] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug A clear and concise description of what the bug is.
Ragas version: Python version:
Code to Reproduce Install latest version of ragas using git
attempt from datasets import Dataset from ragas.metrics import context_recall from ragas import evaluate
data_samples = { 'question': ['When was the first super bowl?', 'Who won the most super bowls?'], 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'], 'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']], 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times'] } dataset = Dataset.from_dict(data_samples) score = evaluate(dataset,metrics=[context_recall]) score.to_pandas()
Error trace
Expected behavior A clear and concise description of what you expected to happen.
Additional context Evaluating: 0% 0/2 [00:00<?, ?it/s] Exception in thread Thread-14: Traceback (most recent call last): File "/home/user/miniconda3/envs/textgen/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/executor.py", line 95, in run results = self.loop.run_until_complete(self._aresults()) File "/home/user/miniconda3/envs/textgen/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/executor.py", line 83, in _aresults raise e File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/executor.py", line 78, in _aresults r = await future File "/home/user/miniconda3/envs/textgen/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one return f.result() # May raise f.exception(). File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/executor.py", line 37, in sema_coro return await coro File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/executor.py", line 111, in wrapped_callable_async return counter, await callable(*args, kwargs) File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/metrics/base.py", line 132, in ascore raise e File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/metrics/base.py", line 125, in ascore score = await asyncio.wait_for( File "/home/user/miniconda3/envs/textgen/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/metrics/_context_recall.py", line 169, in _ascore results = await self.llm.generate( File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/llms/base.py", line 93, in generate return await agenerate_text_with_retry( File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped return await fn(*args, *kwargs) File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in call do = self.iter(retry_state=retry_state) File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/tenacity/init.py", line 314, in iter return fut.result() File "/home/user/miniconda3/envs/textgen/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.get_result() File "/home/user/miniconda3/envs/textgen/lib/python3.10/concurrent/futures/_base.py", line 403, in get_result raise self._exception File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/tenacity/_asyncio.py", line 50, in call result = await fn(args, kwargs) File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/llms/base.py", line 170, in agenerate_text return await self.langchain_llm.agenerate_prompt( File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 687, in agenerate_prompt return await self.agenerate( File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 647, in agenerate raise exceptions[0] File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 832, in _agenerate_with_cache result = await self._agenerate( File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/langchain_openai/chat_models/base.py", line 623, in _agenerate response = await self.async_client.create(messages=message_dicts, **params) File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 1181, in create return await self._post( File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/openai/_base_client.py", line 1790, in post return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls) File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/openai/_base_client.py", line 1493, in request return await self._request( File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/openai/_base_client.py", line 1584, in _request raise self._make_status_error_from_response(err.response) from None openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-NkLIY***lyRC. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
ExceptionInRunner Traceback (most recent call last) Cell In[153], line 13 5 data_samples = { 6 'question': ['When was the first super bowl?', 'Who won the most super bowls?'], 7 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'], (...) 10 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times'] 11 } 12 dataset = Dataset.from_dict(data_samples) ---> 13 score = evaluate(dataset,metrics=[context_recall]) 14 score.to_pandas()
File /home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/evaluation.py:255, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map) 252 if not evaluation_group_cm.ended: 253 evaluation_rm.on_chain_error(e) --> 255 raise e 256 else: 257 result = Result( 258 scores=Dataset.from_list(scores), 259 dataset=dataset, 260 binary_columns=binary_metrics, 261 )
File /home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/ragas/evaluation.py:237, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map) 235 results = executor.results() 236 if results == []: --> 237 raise ExceptionInRunner() 239 # convert results to datasetlike 240 for i, in enumerate(dataset):
ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass
raise_exceptions=Falseincase you want to show only a warning message instead.