search

ShishirPatil / gorilla

Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
https://gorilla.cs.berkeley.edu/
Apache License 2.0
11.51k stars 1.01k forks source link

[BFCL] Add support for Writer models and Palmyra X 004 #755

Open samjulien opened 1 week ago

samjulien commented 1 week ago

This PR adds support for Writer models and our latest Palmyra X 004 to BFCL. Thank you!

samjulien commented 1 week ago

Thanks for the PR @samjulien !

I noticed that here, you intentionally removed all the response field from the function doc. For model that doesn't support response field, we add those information to the end of the function description because we don't want information loss (code here). Is there a reason why you chose to do it that way?

Ah thank you! I’ll add a check for our models there and remove from our handler.

samjulien commented 16 hours ago

Hi @HuanzhiMao, I made that change you suggested and rebased the branch to be updated with main (note: force push), but I'm getting some weird errors now that I wasn't getting before these updates. Looks like it is happening in multi_threaded_inference, getting KeyError: 'function'. I'll keep investigating, but maybe you can take a look/run it locally? Or have you seen this with other providers recently? Anything to go on would be helpful. Thanks!

HuanzhiMao commented 12 hours ago

Hey @samjulien, I have submitted a PR to your fork. https://github.com/samjulien/gorilla/pull/1

Two things I changed:

  1. All the methods ending with _prompting are removed, because they are never used. Palmyra is a Funciton Calling model, and so only those _FC methods will be used.
  2. I made the WriterHandler inherit from the OpenAIHandler since they share the exact same logic for decoding, tool handling, etc. The only difference is _query_FC because writerai use self.client.chat.chat instead of self.client.chat.completions.create

I have tested the code locally and it is working perfectly fine.