Support Interleaved Function Calling

[lapp0]

What behavior of the library made you think about the improvement?

Generation cannot be interleaved with function calls and return values.

How would you like it to behave?

We should allow users to augment LLMs with OpenAI-style function calls.

This would be a great integration for Outlines for the following reasons:

• 1) Outlines is designed to enhance the correctness of language model generations. This is in pursuit of that same goal through different means.
  • a. LLMs are bad at generating consistent output structure (addressed by outlines currently)
  • b. LLMs are bad at math (can be addressed through funciton calling)
  • c. LLMs suffer from hallucinations and missing information (can be addressed through function calling)
• 2) Outlines already implements many of the necessary components and is structured well for function calling.
  • a. Guaranteeing function arguments follow a predefined schema
  • b. Performing post-processing steps after model generates logits
• 3) This is a highly sought after, but poorly supported feature, heavily used by OpenAI. It would be smart for Outlines to become the standard bearer for its use within the open LLM space

Proposed Structure

We should remain flexible in implementation, as some models may be finetuned for a different format. However we should set a default standard which is easily capturable by regex:

Allow generation of (anything other than ```<function_name>) ( ```<function_name> followed by function kwarg confirming JSON) repeating

After function call, we update the generation with the functions return value.

Structure Example:

### System:
You are a language model equipped with function calling capabilities. 
These functions can provide you with additional information to help you generate a response.
You may call a function while generating output by typing
 ```<function_name>
<json object representing function arguments>``` 

You have the following functions available:
- get_weather(longitude: float, latitude: float)
- calculate(mathematical_expression: str)

### User:
I have 15 apples, my brother ate 2, my sister ate 3, and I picked 4 more. How many apples do I have

### Assistant:
```calculate
{"mathematical_expression": "15-2-3+4"}
``

At this point, calculate is called, and the response is injected:

```result
14
``

Then generation proceeds:

You have 14 apples

Code Example

from sympy import sympify
def calculate(expr: str) -> str:
     return sympify(expr)

import outlines

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

# this interface is up for debate, maybe we want outlines.generate.interleaved_function
generator = outlines.generate.text(model, callable_functions=[calculate, get_weather])

result = generator(prompt)

Implementation

To resolve this issue we need to implement the following

• List[Callable] to function calling regex pattern, enabling free generation of text until the function call begins (```<function_name>)
• Updated SequenceGenerator which can intercept function calls and perform them
• In some cases we will be able to directly inject the function results tokens into the generation, but for some models we need to constrain the generation such that only the function results are produced before moving on.
• Documentation: It's important we explain to the user that the model should be capable of understanding function calling, and the prompt be designed to handle function calling.

References

Prior art:

• https://github.com/rizerphe/local-llm-function-calling
• https://docs.llama-api.com/essentials/function
• https://github.com/OpenBMB/ToolBench
• https://github.com/MeetKai/functionary

Useful tool:

• https://github.com/KillianLucas/open-interpreter/

Models:

• Code models such as WizardCoder and StarCoder are likely the best tools.
• https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling (proprietary)
• https://huggingface.co/Trelis/Mistral-7B-Instruct-v0.1-function-calling-v2 (proprietary)

[jeffreyftang]

Hi @lapp0, definitely interested in this feature! Curious if this is something that's planned for the near term?

[lapp0]

I'm a bit busy the coming weeks, but I'd be happy to provide guidance if you, or anyone else is interested in implementing!

Having this implemented first would help a lot:

https://github.com/outlines-dev/outlines/issues/667

[jeffreyftang]

Happy to contribute! Before I commit, though, I'd like to get a sense of the scope of work and estimated time involved. I don't want to make promises I can't keep :)

[lapp0]

Great to hear @jeffreyftang

I think we should focus on https://github.com/outlines-dev/outlines/issues/667 for now. If you're interested in implementing that component, could you please ping remi in that issue?

Once that part is implemented, interleaved function calling really just involves using #667 to implement the loop described in this issue and enable function

• [ ] #667
• [ ] Write base class which accepts a list-hinted of typed functions
• [ ] write method which converts each function signature -> a json schema -> an interegular.FSM, then modify to "Allow generation of (anything other than <function_name>) ( followed by function kwarg confirming JSON) repeating" (tooling to accomplish this already exists in Outlines)
• [ ] Write method which extracts function arguments from the generated text, calls the function, and injects the result, and repeats the loop
• [ ] Handle EOS tokens