Closed Data-drone closed 4 months ago
I got things working by just running format()
against the prompt my LLM received if it was in ChatPromptTemplate
format but not sure that is the best approach.
I have set the chat_qa=0.0
so shouldn't all the prompts be coming in as string?
Hey @Data-drone , can you share the ragas & python version you're using?
ragas==0.0.18 python 3.10.12
Hi @Data-drone , We will look into the langchain custom llms issue. Regarding setting chat_qa=0
, it is used to control if the test set should contain conversational questions or not.
Hi @Data-drone , which model are you using? Are you using an instruction-tuned model (that is not chat based)?
Vicuna 1.5 13b
hey @Data-drone , so we do have this snipped inside RagasLLMs that should handle this logic for you
if isinstance(self.llm, BaseLLM):
ps = [p.format() for p in prompts]
result = self.llm.generate(ps, callbacks=callbacks)
else: # if BaseChatModel
ps = [p.format_messages() for p in prompts]
result = self.llm.generate(ps, callbacks=callbacks)
if your custom model is a base class of either, it should do the conversation for you. but not sure why its not working here. Could you show us how you're calling the testsetgenerator and custom models code?
@shahules786 @jjmachan In the same context, TestsetGenerator uses cross-encoder/stsb-TinyBERT-L-4/ from hugging face. Is there a way, i can avoid using that and just use gpt models from azure open ai alone and generate synthetic dataset?
hey @msunkarahend thanks for bringing that up. what you can do is change it via the class
class TestsetGenerator:
"""
Ragas Test Set Generator
Attributes
----------
generator_llm: LangchainLLM
LLM used for all the generator operations in the TestGeneration paradigm.
critique_llm: LangchainLLM
LLM used for all the filtering and scoring operations in TestGeneration
paradigm.
embeddings_model: Embeddings
Embeddings used for vectorizing nodes when required.
chat_qa: float
Determines the fraction of conversational questions the resulting test set.
chunk_size: int
The chunk size of nodes created from data.
test_distribution : dict
Distribution of different types of questions to be generated from given
set of documents. Defaults to {"easy":0.1, "reasoning":0.4, "conversation":0.5}
"""
def __init__(
self,
generator_llm: RagasLLM,
critic_llm: RagasLLM,
embeddings_model: Embeddings,
testset_distribution: t.Optional[t.Dict[str, float]] = None,
chat_qa: float = 0.0,
chunk_size: int = 1024,
seed: int = 42,
) -> None:
so if you give a new different Embedding
instance you will be able to change it.
let me know if you want the finished snippet and I'll share that too 😄
but @shahules786 probably we should change from_defaults
class method to with_openai
or something like that to make it easier?
@jjmachan If you can share the finished snippet for the TestsetGenerator, it would be great. thanks in advance.
I notice that the if downloaded from source, the version of ragas on github is 0.0.23.dev44+g506ad60, while the latest version I down load from pip is 0.0.22, same as shown "latest realease" on github. And it seems like 0.0.22 will meet similar question about list(str)
and list(ChatPromptTemplate)
.
Could you guys try #670?
Hi Team,
I am trying to get the
TestsetGenerator
to work properly with a Langchain custom LLM.It's
_call
method current expects a string prompt but it gets aChatPromptTemplate
from ragas. what is the best way to handle this?