search

mudler / LocalAI

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed inference
https://localai.io
MIT License
23.1k stars 1.75k forks source link

feature: ChatML support #67

Open limcheekin opened 1 year ago

limcheekin commented 1 year ago

Hi there,

Thanks for sharing the codes.

May I know any plan to support ChatML?

You can find out more about ChatML with the following links: https://github.com/openai/openai-python/blob/main/chatml.md https://cobusgreyling.medium.com/the-introduction-of-chat-markup-language-chatml-is-important-for-a-number-of-reasons-5061f6fe2a85

Thanks.

localai-bot commented 1 year ago

:warning::warning::warning::warning::warning:

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

_but.... I can also be funny or helpful :smilecat: and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me, I don't support (yet) replying!

:warning::warning::warning::warning::warning:

ChatML is a proposed standard for representing chatbots and their interactions in a structured way. It allows for easier development, testing, and deployment of chatbots across different platforms and languages. LocalAI currently supports GPT-2 based text generation using llama.cpp and other backends. However, we are always looking to improve our capabilities and expand our support for new technologies and standards. We will consider adding ChatML support in the future depending on demand and feasibility.

Sources:

no-skynet commented 11 months ago

Having found models from TheBloke to be particular well documented and easy to work with, I have basically stopped looking at any other. I even built my own TheBloke filtered gallery.

This seemed to work well, for a while at least. However, The Bloke recently switched to using ChatML for all future models.

See here: https://www.google.com/search?hl=en&q=%22All%20of%20my%20future%20models%20will%20use%20ChatML%20prompt%20format%22 "All of my future models will use ChatML prompt format" - TheBloke

ps. I'm not a Go developer, otherwise this would have been a PR to add support for "ChatML" as a chat format. Love this project, and I will be happy to help in any other way I can. I can do Python, Rust, Bun, or anything that doesn't involve code also :-)

mudler commented 11 months ago

Having found models from TheBloke to be particular well documented and easy to work with, I have basically stopped looking at any other. I even built my own TheBloke filtered gallery.

This seemed to work well, for a while at least. However, The Bloke recently switched to using ChatML for all future models.

See here: https://www.google.com/search?hl=en&q=%22All%20of%20my%20future%20models%20will%20use%20ChatML%20prompt%20format%22 "All of my future models will use ChatML prompt format" - TheBloke

ps. I'm not a Go developer, otherwise this would have been a PR to add support for "ChatML" as a chat format. Love this project, and I will be happy to help in any other way I can. I can do Python, Rust, Bun, or anything that doesn't involve code also :-)

the chat format is wide open and allows to template per-message. See my reply in https://github.com/go-skynet/LocalAI/issues/1163#issuecomment-1758129781 - there is an example for llama2 which follows a similar approach.

JFY LocalAI can have backends in any language, for instance there are many in python as well

no-skynet commented 11 months ago

Thank you, @mudler . If I understand correctly and there'll be no code change required, then I suppose we can perhaps close this issue with a response to @limcheekin's question?