search

amiantos / inneal

Inneal is an LLM chatbot/roleplay client for the AI Horde, built with SwiftUI and SwiftData
GNU General Public License v3.0
19 stars 4 forks source link

Context Templates #36

Open amiantos opened 1 month ago

amiantos commented 1 month ago

So far this adds Llama 3 instruct context template in a very hacky way. After implementing it and experimenting with trying to figure out which Horde models are available use it, I felt like it wasn't really worth it. Context templates is very much an advanced user feature that should, hopefully, go away, as LLMs progress. If the need doesn't go away, then I should consider implementing context templates in more robust way similar to Ollama/SillyTavern and not half-assing it like what this PR looks like currently.

Either way, I think I am going to hold off on this and not include it as part of 1.1. I also think I am going to revert some last minute changes to the default context template...

amiantos commented 1 month ago

I asked on the kobold horde channel about which llama 3 models actually use the instruct template. It seems like most do, but some do not, and one even uses ChatML (it has chat in the name). So, the problem is hard to overcome without building our own internal database of which models use which format. Which could be fun. But feels very out of scope for the goals of this project...