search

SciSharp / LLamaSharp

A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.
https://scisharp.github.io/LLamaSharp
MIT License
2.46k stars 331 forks source link

Is it possible, that a model ist also able to use the online search during a chat? #291

Open hswlab opened 9 months ago

hswlab commented 9 months ago

I assume it is not possible for a locally running chatbot to perform an online search during a conversation to retrieve current information, e.g. weather or news?

martindevans commented 9 months ago

That's probably something that would be built into an app on top on LLamaSharp, rather than directly into LLamaSharp itself.

SignalRT commented 9 months ago

@hswlab, You could see LLamaSharp as the component that manages the LLM. It's easy to make that kind of things if you use Semantic Kernel in top of LLamaSharp. SK is designed to manage the integration of functions, plugins, etc.

Documentation: https://learn.microsoft.com/en-us/semantic-kernel/overview/

Something similar of what you are looking for: https://www.youtube.com/watch?v=q2c5gaLTFag

ynanech commented 9 months ago

I am also very interested in implementing the plugin function of llama2, similar to the ChatGPT plugin, which is very practical

martindevans commented 9 months ago

Do you think there are any features that LLamaSharp would need to support for you to build plugin/function calling?

I think using a Grammar and appropriate prompting should do the job?

stephentoub commented 9 months ago

Do you think there are any features that LLamaSharp would need to support for you to build plugin/function calling?

SK now passes the Kernel into the connector, giving the connector access to all the plugins. This makes it possible for the connector to support automated function calling. The SK OpenAI-based implementation now does this... it'd be awesome if LlamaSharp could as well, if using a model fine-tuned for function calling. Alternatively, the connector could implement it via a separate planning step, still as part of the chat call from the client.

AsakusaRinne commented 9 months ago

@stephentoub If I didn't misunderstand, you're saying LLamaSharp should provide a wrapper to make it possible to execute multi-step call depending on a rule? I'm not familiar to SK...I'm wondering whether it's expected to be done in LLamaSharp or the integration with SK.

stephentoub commented 9 months ago

@stephentoub If I didn't misunderstand, you're saying LLamaSharp should provide a wrapper to make it possible to execute multi-step call depending on a rule? I'm not familiar to SK...I'm wondering whether it's expected to be done in LLamaSharp or the integration with SK.

I'm saying some models support function calling, where you can pass to the model a list of functions/tools you'd like it to be able to use optionally, and as part of its response it can ask you to invoke one or more of those functions with a particular set of arguments it provides. You can do so manually, with or without SK, but SK also makes it possible for the IChatCompletion implementation to handle that automatically: the caller passes in the Kernel containing the available plugins, and the IChatCompletion implementation can in turn pass that function information along to the model, and if the model asks for a function to be invoked, the IChatCompletion implementation can do the invocation and give the results back to the model, so that the caller doesn't have to. Here's an example with the OpenAI connector: https://github.com/microsoft/semantic-kernel/blob/ffea465c05d0d9d53a2277d16ee6e969f279f4e6/dotnet/samples/KernelSyntaxExamples/Example59_OpenAIFunctionCalling.cs

AsakusaRinne commented 9 months ago

I'm saying some models support function calling, where you can pass to the model a list of functions/tools you'd like it to be able to use optionally, and as part of its response it can ask you to invoke one or more of those functions with a particular set of arguments it provides. You can do so manually, with or without SK, but SK also makes it possible for the IChatCompletion implementation to handle that automatically: the caller passes in the Kernel containing the available plugins, and the IChatCompletion implementation can in turn pass that function information along to the model, and if the model asks for a function to be invoked, the IChatCompletion implementation can do the invocation and give the results back to the model, so that the caller doesn't have to. Here's an example with the OpenAI connector: https://github.com/microsoft/semantic-kernel/blob/ffea465c05d0d9d53a2277d16ee6e969f279f4e6/dotnet/samples/KernelSyntaxExamples/Example59_OpenAIFunctionCalling.cs

Thank you! It seems that we should support it in the integration with Sk for convenience. The example above makes sense. FYI @xbotter

MrCSharp22 commented 6 months ago

Hi @AsakusaRinne has there been any progress on this? Is there a better place to track the progress of integrating LlamaSharp with SK?

kosimas commented 6 months ago

@MrCSharp22 I'm currently working on a solution in my fork. You can find a working example here https://github.com/kosimas/LLamaSharp/blob/master/LLama.Examples/Examples/SemanticKernelFunctionCalling.cs For the example you have to use this model https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v3

The implementation is a bit dirty and not complete. I will create a pull request a soon as I'm happy with my code and no one else has come with a better solution. I'm also open for ideas and improvements.

MrCSharp22 commented 6 months ago

Thanks @kosimas that is working great! I was trying a similar approach but was using v2 of that model. Would love to see this brought into upstream LlamaSharp.

MithrilMan commented 3 months ago

any update on this?