Open landhu opened 1 month ago
Hey @landhu! I'm here to help you out with any bugs, questions, or contributions you might have. Let's tackle this together!
To specify the name of the model and avoid the "model 'gpt-3.5-turbo' not found" error when using a locally deployed LLM with the OpenAI interface, you can modify the model
parameter in the llm_factory
function. Here is the relevant code snippet:
def llm_factory(
model: str = "gpt-3.5-turbo", run_config: t.Optional[RunConfig] = None
) -> BaseRagasLLM:
timeout = None
if run_config is not None:
timeout = run_config.timeout
openai_model = ChatOpenAI(model=model, timeout=timeout)
return LangchainLLMWrapper(openai_model, run_config)
To specify a different model name, simply pass the desired model name as the model
argument when calling llm_factory
. For example:
llm = llm_factory(model="your-local-model-name")
This will configure the ChatOpenAI
instance to use the specified model name.
Additionally, if you are using Langchain, you can pass the Langchain LLM instance directly and Ragas will wrap it with LangchainLLMWrapper
. For example:
from langchain_together import Together
together_key = "<your-key-here>"
together_completion = Together(
model="NousResearch/Nous-Hermes-2-Mixtral-8x7B-SFT",
temperature=0.7,
max_tokens=4000,
top_k=1,
together_api_key=together_key
)
llm = llm_factory(model=together_completion)
This approach ensures that your custom LLM is properly integrated and avoids the "model not found" error [1][2].
To continue talking to Dosu, mention @dosu.
Thanks Very much. If I just want evaluate like simple demo: ` from datasets import Dataset import os from ragas import evaluate from ragas.metrics import faithfulness, answer_correctness
os.environ["OPENAI_API_KEY"] = "your-openai-key"
data_samples = { 'question': ['When was the first super bowl?', 'Who won the most super bowls?'], 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'], 'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']], 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times'] }
dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset,metrics=[faithfulness,answer_correctness]) score.to_pandas()`
How Can I pass the mode parameter?
To pass the model parameter for evaluation in your example, you can follow these steps:
Host the custom model using VLLM (if not already done):
# Run this on your terminal with GPU enabled
python -m vllm.entrypoints.openai.api_server --model your-local-model-name
Set up the ChatOpenAI
instance with the custom model:
from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness
from ragas.llms import LangchainLLMWrapper
from langchain.chat_models import ChatOpenAI
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
inference_server_url = "http://localhost:8000/v1"
MODEL = "your-local-model-name"
chat = ChatOpenAI(
model=MODEL,
openai_api_key=os.getenv("OPENAI_API_KEY"),
openai_api_base=inference_server_url,
max_tokens=2048,
temperature=0,
)
langchain_llm = LangchainLLMWrapper(chat)
data_samples = {
'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'],
['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}
dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset, metrics=[faithfulness, answer_correctness], llm=langchain_llm)
print(score.to_pandas())
This code sets up the custom model to be used for evaluation instead of the default model [1][2].
To continue talking to Dosu, mention @dosu.
@landhu where you able to fix this?
[ ] I checked the documentation and related resources and couldn't find an answer to my question.
How do I specify the name of the model An LLM is deployed locally. All interfaces are the same as openai, but the mode name is different
Code Examples Now I can do this in the following way to change key and url
os.environ['OPENAI_API_KEY'] ="xxxx" os.environ['OPENAI_API_BASE'] = "xxxx"
But after that, when I run simple code: `data_samples = { 'question': [ 'When did Einstein born?', 'Where did Einstein born?', ], 'answer': [ 'Einstein was born in 1879.', 'Einstein was born in Germany.','Einstein was born in 1879 in Germany.'
} dataset = Dataset.from_dict(data_samples) score = evaluate(dataset, metrics=[answer_correctness]) print( score.to_pandas() )`
it's shows: openai.NotFoundError: Error code: 404 - {'error': {'message': "model 'gpt-3.5-turbo' not found, try pulling it first", 'type': 'api_error', 'param': None, 'code': None}}