Closed ahuang11 closed 3 months ago
Hello @ahuang11! I'm Dosu, a bot here to lend a hand with bugs, answer questions, and guide you towards becoming a contributor while we wait for a human maintainer to step in. I'm diving into your issue now and will get back to you with a detailed answer shortly.
Understood. Use instructor.patch
for now.
And we'll need to make a PR to make sure base York exists first before we check it.
@jxnl Can you share sample code? I'm getting the same error.
To work around the AttributeError: 'Llama' object has no attribute 'base_url'
issue, you can apply a temporary patch in your code. This involves manually setting the base_url
attribute on the Llama
object or modifying the usage to avoid accessing base_url
directly. Since the maintainer suggested using an instructor patch, here's a basic example of how you might implement a workaround:
# Assuming you have your Llama object initialized as `llama`
# Directly setting the base_url attribute as a workaround
llama.base_url = "your_base_url_here"
# Proceed with your code logic...
Please replace "your_base_url_here"
with the actual base URL you intend to use, if applicable. This is a temporary fix until the official patch is released. Keep an eye on the repository for updates or a permanent solution to this issue.
To continue the conversation, mention @dosu.
the base_url issue will be fixed in 1.2.3
Thanks for the update.
I think there's still an assert statement blocking this:
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[6], line 28
13 model_path = hf_hub_download(
14 "TheBloke/OpenHermes-2.5-Mistral-7B-GGUF",
15 "openhermes-2.5-mistral-7b.Q4_K_M.gguf",
16 )
17 llama = llama_cpp.Llama(
18 model_path=model_path,
19 n_gpu_layers=-1,
(...)
26 verbose=False,
27 )
---> 28 create = instructor.from_openai(
29 client=llama,
30 mode=instructor.Mode.JSON_SCHEMA, # (2)!
31 )
32 message = {"role": "user", "content": "Teach me how to say `Hello` in three languages!"}
33 extraction_stream = create(
34 response_model=instructor.Partial[Translations], # (3)!
35 messages=[message],
36 stream=True,
37 )
File ~/miniconda3/envs/panel/lib/python3.10/site-packages/instructor/client.py:277, in from_openai(client, mode, **kwargs)
274 else:
275 provider = Provider.OPENAI
--> 277 assert isinstance(
278 client, (openai.OpenAI, openai.AsyncOpenAI)
279 ), "Client must be an instance of openai.OpenAI or openai.AsyncOpenAI"
281 if provider in {Provider.ANYSCALE, Provider.TOGETHER}:
282 assert mode in {
283 instructor.Mode.TOOLS,
284 instructor.Mode.JSON,
285 instructor.Mode.JSON_SCHEMA,
286 instructor.Mode.MD_JSON,
287 }
AssertionError: Client must be an instance of openai.OpenAI or openai.AsyncOpenAI
im dumb im so sorry.
i need to add tests for this.
Thank you! I am trying to use llama
with Instructor as well to get structured outputs. Are you suggesting that using instructor.patch
is the best way to go forward at the moment?
@amaarora if ur running llama on groq, this worked for me
groq_client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
client = instructor.from_groq(groq_client, mode=instructor.Mode.JSON)
chat_completion = client.chat.completions.create(
messages=[
{
"role": "system",
"content":sys_prompt,
},
{
"role": "user",
"content":user_prompt,
},
],
model="llama3-70b-8192",
temperature=temp,
max_retries=Retrying(
stop=stop_after_attempt(2),
wait=wait_fixed(60), # rate limit on groq lol
),
response_model=self.response_model,
)
What Model are you using?
Describe the bug A clear and concise description of what the bug is.
The documented example doesn't work https://github.com/jxnl/instructor/blob/main/docs/hub/llama-cpp-python.md?plain=1
To Reproduce Steps to reproduce the behavior, including code snippets of the model and the input data and openai response.
I tried to update it:
AttributeError: 'Llama' object has no attribute 'base_url'
Expected behavior A clear and concise description of what you expected to happen.
It should not raise an error about
base_url
Screenshots If applicable, add screenshots to help explain your problem.