cyrillic doesn't work

[00jeser]

I have model which generating text using cyrillic alphabet. It's work in llama-cpp-python but in LLamaSharp I heve unknown symbols: Here is my code:

var modelPath = @"C:\Temp\ggml-saiga-13b-q4_1.bin";
var model = new LLamaModel(new LLamaParams(model: modelPath, n_ctx: 512, interactive: true, antiprompt: File.ReadAllLines("antiprompt.txt").ToList(),
                repeat_penalty: 1.0f));
var session = new ChatSession<LLamaModel>(model).WithPromptFile("prompt.txt").WithAntiprompt(File.ReadAllLines("antiprompt.txt"));
var outp = session.Chat("User: Почему трава зеленая? \r\nSaiga: ", encoding: "UTF-8");
string resp = "";
foreach (var output in outp)
{
    resp += output;
    if (resp.EndsWith(USER_FLAG))
    {
        break;
    }
}

I tried using encodings UTF-8, ASCII, and Unicode, but all of them give same result

[AsakusaRinne]

Hi, could you please provide a way to download your model? I guess it's a model with another language rather than English.

[00jeser]

Yes It is model whith russian language. I downloaded it from hugging face https://huggingface.co/IlyaGusev/saiga_13b_lora_llamacpp/

[AsakusaRinne]

Hi, maybe I find the reason of this problem but I'm not sure because I can't speak Russian at all. So I need your help.

Considering Почему трава зеленая? as an input, is the following text a reasonable answer?

Трава зеленая из-за того, что она содержит большое количество хлорофилла. Это пигмент, который отвечает за зеленую окраску растений. Хлорофилл прекрасно поглощает свет в дополнение к своим отражательным свойствам, которые способны отразить сине-фиолетовые лучи света. Это приводит к зеленому цвету траве. Кроме того, хлорофилл играет важную роль в фотосинтезе, процессе, который позволяет растениям использовать солнечный свет для превращения воду, углекислого газа и питательных веществ в форму, которую они используют для роста. Хлорофилл способствует эффективному проведению этой процедуры.

[00jeser]

Yes, it's correct, but in my case, the answer consists of unidentified characters, not letters like yours

[AsakusaRinne]

Yes, it's correct, but in my case, the answer consists of unidentified characters, not letters like yours

The master branch does have a BUG because of difference of encoding between C# and cpp. I'll submit the fix later.

[00jeser]

ok, thank you a lot

[AsakusaRinne]

I've updated the master branch. Could you please clone the repo and have a try? If everything goes well, the fix will be included in the next release.

[00jeser]

Yes, it's works now

[Xsanf]

Sorry, I put this post in the wrong thread at the beginning.

I included this library in Unity 2021.3.22f1 Since Unity hasn't had time to migrate to later versions of NET, a few changes had to be made.

LLamaParams.cs replacing struct with class because memory allocation has changed. public struct LLamaParams > public class LLamaParams

LLamaTypes.cs - Added

using System.ComponentModel; namespace System.Runtime.CompilerServices { [EditorBrowsable(EditorBrowsableState.Never)] internal static class IsExternalInit { } } Because earlier versions don't have this definition. I did not set the compilation conditions, because I did it in haste.

So that the model can be run under Unity natively. I ran into this error too, thanks for the quick solution. I'm sending this message because I think using LLM in Unity will be in demand for game development.

LLM does not have to be used for dialogues. They are great for spawning events. When populating a rigid logical structure by requesting instructions with examples in context. This allows for non-linear plot generation.

[AsakusaRinne]

Thank you for giving this feedback. We'd like to add Unity support because it's an important application of LLM. We are now refactoring LLamaSharp to fix a BUG and improve the APIs. However I have little experience with unity and don't know how to make it work well in unity. Would you mind telling something about what should we do to make LLamaSharp work in unity directly? We'll really appreciate that. :)

[Xsanf]

I will gladly help. True, it’s worth considering, I’m 67 years old, I have 8 grandchildren and granddaughters)) Which require my attention)) I taught at the institute and now retired)) This is to the fact that I’m not ready for some kind of regular work or support and just drive with what I love, so that the brain does not fall asleep)) I am ready to pass on what I did or conduct some tests. But, unfortunately, I’m not well adapted to systematic work))

How can I help? At the very least, can I make a unity package out of this with an example that I can just import into Unity and see the example? Unity is just convenient with a very low entry threshold.

Just a minimal general understanding of the concept of component programming. At any point in time you are working with the scene (simplified, what you see on the screen) The whole program is a windowsmessage processing loop (embedded in the scene) into which the components you created are embedded. Since new components can be added to those already present in the scene, it is possible to create components that are complex hierarchies of other components, making it easier to create and use complex aggregations. The order in the hierarchy creates call sequence rules. The common scene manager makes it easy to find individual components or their collections, which allows them to be linked later and build complex event or data management logic. For convenience, a component from the stage can be dragged into the Prefabs directory, turning it into a Prefab, simply an automatically generated template that can be propagated by dragging it to any stage or created programmatically via a factory call. Components inherit from the standard MonoBehaviour class. Where two main methods are used // Start is called before the first frame update voidStart() {} // Update is called once per frame void Update() {} The first one is called at the first start and usually serves just to initialize the component and other primary settings. The second one is called when generating each image frame on the stage, so it is undesirable to place long-executing code in it. As a rule, such code is executed by Coroutines or other components are created that do not need to process each frame, but immediately implement their own endless processing cycle. With whom you can contact either through events or through a subscription. This is the main logic, and the rest is quite within the framework of general C # programming. Of course, there are a lot of delicious things, but any information and examples are easily available.

The biggest problem for you is that Unity is multi-platform. Therefore, he has a need for the simultaneous development of compatible libraries for different platforms. The addition of NET libraries is lagging behind in order to maintain compatibility. The latest version does not work in Unity NET 4.2. (They promise support for NET 6.0 in 1-2 years) But this is not a serious problem. If something works under Visual Studio C#, then it can usually be run on Unity as well. True, this may already lose its multiplatform nature. The difference in NET versions is not big and therefore, with minimal changes that I described, everything works fine under Unity. A lot of Warning is issued. But in fact, the NET version descriptions state that the features have not been changed and these warnings are simply due to a formal version mismatch. I used Unity version 2021.3.22f1. no coincidence since earlier versions are based on NET 2.X

Another feature, since components can run in the environment without starting the project, the body of the component splits (optionally) into two files. Directly runtime and Unity editor. This allows for each component to create specific input forms and wizards that greatly simplify the work with the components. In the editor, the component can work in two modes. Not working(runtime only), Edit mode. The current status(Editor, runtime) is available to the compiler.

[Xsanf]

I do not want to strain you, because I can imagine the amount of work you are doing. But there is a problem that is not emphasized in the community. It is common knowledge that LLM models are divided into left-hand context and two-sided models, like GPT and BERT. It is no less known that GPT is more convenient for generation. Now there are T5, which in the learning process work as two-way, but have the same advantages in generation as GPT (although the overhead for training is higher). In fact, it rests on the problem of languages. The learning process, if not focused on the mathematical techniques themselves, implements the idea of LSA (Latent Semantic Analysis) analysis of nearby events in the event horizon window. The language itself is a very curious phenomenon. This is the ability, evolutionarily developed by the brain, to translate its highly parallel state of displaying the world into a sequence of detonats, in accordance with the current attention and the width of the event horizon window. In linguistics, a detonate is a sign associated with a concept. A concept is a set of certain states (for the brain as well as for the neural network of connections between neurons), which has some semantics, which is determined by connections with other detonats. The process of learning LLM is extraction the structure of concepts, in the process of analyzing the detonations of the texts presented for learning. Since the personality of a person is a purely verbal phenomenon, we can say that we are recreating a model of personality based on texts. In the case of LLM, this is immediately obtaining all the archetypes displayed in the training texts. The phrase often reported by AI that it is a statistical model is absolutely not true. That is why weights are called weights and not probabilities))

I would like to draw your attention to the T5 model. If English, like some others, is projective, i.e. the context necessary for understanding the word will always be on the right, then Russian and many other languages ​​are not projective, both right and left contexts are needed to understand the word. Of course, in non-projective languages, not all sentences are non-projective, and some information can be extracted from a non-projectively constructed sentence. But this does not negate the fact that GPT will extract less information from the text in Russian than T5.

I would like to ask you, to the extent possible, to connect T5 models to your project, at the moment these models will be developed by Sberbank and Yandex. They promise to release these models for personal use by the end of this year or later. Line of standard sizes. ai-forever/FRED-T5-1.7B is now available on Hugging Face

[Xsanf]

I put a working Unity example at https://github.com/Xsanf/LLaMa_Unity. Just install Unity 2021 or higher and create an empty 3D project. Assets > Import package > Custom package "LLaMa_Unity.unitypackage". After installing it. In Project Panel > Scene. Load SampleScene. Set the unsafe flag, and run the example.

[AsakusaRinne]

Hi, thank you a lot for your great work! Apologies on the late reply because what you said is valuable and new for me, therefore it took some time for me to understand and have simple practices before reaching you further. These two days I was configuring the unity environment and learning T5 model while refactoring LLamaSharp.

I am ready to pass on what I did or conduct some tests. But, unfortunately, I’m not well adapted to systematic work))

I fully understand you, I'd be grateful for some examples and I'll write the code to make it compatible with unity.

Thanks to your detailed and clear explanation, I think I've known a bit basic knowledges for unity. I'll have a try of your example in unity since I've configured it. There's something about unity that I didn't find an answer and I'll appreciate it if you'd like to help me. Generally, after having installed a game made with unity, there will be a folder containing all the game files, including exe, libraries and assets. However when compiling an unity project, it seems not to have the concept of output directory as visual studio C# does. Unfortunately LLamaSharp.Backend works by copying library files to the output directory in VS. Furthermore it seems unity doesn't encourage users to install packages via nuget because "everything in unity is resource". Therefore the only way I can think of is to download the dll and add it to the resource manually, which is a bit inconvenient. In unity is there a way to make this process as convenient as in VS?

I would like to ask you, to the extent possible, to connect T5 models to your project, at the moment these models will be developed by Sberbank and Yandex.

I will but it won't be quick because llama.cpp hasn't supported it till now. I tried T5 small model with llama.cpp and error occurred when converting model to ggml format. Issue1 and issue2 also provided evidence for it. I'd like to try to add support for it in llama.cpp if no one does so in the future. Besides, it seems that flan-T5 has even better performance than original T5 model.

In C# field, it may also be possible to run T5 model with tensorflow.net, which is also developed by SciSharp. It has already been able to load bert model and run (refer to simple_bert_test). The lack of tokenizer is the main problem but it's on the way now. I'll also try it later.

Thanks again for all your helps! They mean a lot to me. :)

[Xsanf]

I'll clarify right away. I am not a Unity senior. It's just that I don't know a lot of different languages at the same time.)) It's bad in terms of practice. Regarding NuGet, I'll just give a few links that the search for "Unity nuget" gives)) https://stackoverflow.com/questions/53447595/nuget-packages-in-unity https://github.com/GlitchEnzo/NuGetForUnity https://github.com/xoofx/UnityNuGet

Let me remind you that Unity has compatibility issues due to the multiplatform that they solve. But if you do not want to wait two years)) That is, there are two options. After unzipping, include the package in your project manually or use some tool created in Unity. Unity allows you to write your own extension tools in itself, which makes it amazingly convenient (if you learn how to do this, of course) Good packages from the Unity store allow you to turn Unity into anything. For example, in a very convenient professional animation package for cut scenes, integrating it into the interface. So, I'm afraid there is no easy way. But they are not an insurmountable problem either.

We communicate through google translator. I did not quite understand the question about the assembly of the finished project. I just moved the DLL by compiling your project in VS to the Plugins folder, thus including them in the Unity project. I copied your LLamaSharp source code into Unity and got a bunch of errors, just looked at what was causing them and made a minor edit. I suspect this is the easiest way. In general, File > Build setting is responsible for the assembly. Since the entire project will be located either on a single scene, or include many scenes, then all these scenes in the Build setting window, you need to add Scenes in bild to the top box. They are added with the "Add open scenes" button. those. the scene must be open in the scene editor. All specified scenes will be included in the assembly. You will be prompted to choose a save location after clicking "Build". Unity itself will determine, when analyzing scenes, which resources are associated with the project and will place them in the working directory, taking into account all dependencies. Project settings Edit > Project setting. There are a lot of them, but at the beginning you will be interested in "Player" there will be an unsafe flag, a dll connection standard and a NET compatibility level.

About T5, it's just a wish)) I think the models will be in demand outside the English language. GPT is a great approach, but only for projective languages. In general, model size seems to be less important than training sample quality and two-way context in the case of non-projective languages. And of course LoRA, which will carry out the target setting of the model, through attention management. I just don't want to break the law)) Telling here what questions model 13b gave me a detailed answer))) But when discussing the script, she suggested a taxonomy of genres. The name of the genre, a brief description of the essence of the genre and an example of a typical work. There are about 20 lines in total. And all examples of works were real and really were examples of the genre. Those. she didn't hallucinate. Even 13b knows an amazing amount. As AI for games even 3b T5 will be more than enough.

[Xsanf]

The style sheet that the model returned. When we discussed how to set her storytelling style. Alignment (jailbreak) was removed from the model. Shell https://github.com/LostRuins/koboldcpp/releases Model https://huggingface.co/TheBloke/chronos-13B-GGML/resolve/main/chronos-13b.ggmlv3.q5_1.bin The model is focused on creating long texts on which it was trained.

theme	description	example
Gore and violence	Graphic depictions of violence, blood, and gore, with a focus on the horror and brutality of war and conflict.	"Game of Thrones"
Nudity and sexual content	Sex scenes that are explicit and unapologetic, featuring full nudity and realistic sexual encounters.	"Nymphomaniac"
Profanity and crass humor	Frequent use of strong language, including sexually charged and offensive language.	"The Big Lebowski"
social commentary	Thought-provoking commentary on current social issues, such as racism, sexism, and inequality.	"Get out"
Mind-bending plot twists	Shocking and unexpected plot twists that challenge the audience's expectations and understanding of the story.	"The Prestige"
complex characters	Characters that are multi-dimensional and conflicted, with deep backstories and intricate motivations.	"Breaking Bad"
Suspense and tension	Tense and gripping scenes that keep the audience on the edge of their seats, with a focus on building and sustaining suspense.	"The Silence of the Lambs"
Symbolism and metaphor	Use of symbols and metaphors to add depth and meaning to the story, exploring themes and ideas in a more abstract way.	"Eternal Sunshine of the Spotless Mind"
Unconventional storytelling	Unique innovative and approaches to storytelling, with non-linear narratives and unconventional characters.	blade runner
Social groups in conflict	Depiction of social groups in conflict with each other, exploring themes of discrimination, inequality, and prejudice.	"Get out"
Mind-bending visuals	Incredible and surreal visuals that challenge the audience's perceptions and understanding of reality.	"Inception"
Psychological drama	Focus on the psychological and emotional struggles of the characters, with a deep exploration of their thoughts, motivations, and insecurities.	"The Killing"
Emotional depth	Depiction of complex and multifaceted emotions, exploring the full range of human experiences and emotions.	"The Pursuit of Happyness"
black humor	Use of dark humor to explore complex and difficult topics, such as death, illness, and trauma.	"The Big Lebowski"
Conspiracy and paranoia	Depiction of a world filled with conspiracy and paranoia, with a focus on the fear and mistrust that can arise in such a world.	"The Parallax View"
Mind-bending narrative	Complex and nonlinear narrative structure that challenges the audience's understanding of the story and leaves them questioning what is real and what is not.	"Primer"
social commentary	Depiction of social and political issues, and the impact they have on individuals and society.	"The Wolf of Wall Street"
Unreliable narrators	Depiction of characters whose perceptions and understanding of events are called into question, leaving the audience uncertain of what is true and what is not.	"The Usual Suspects"
Noir-inspired	Inspired by the dark, moody, and often violent films noir of the 1940s and 1950s, with a focus on a gritty and realistic portrayal of crime and corruption.	Chinatown
Satire and dark comedy	Use of humor and irony to criticize and mock contemporary social and political issues.	"The Grand Budapest Hotel"
Psychological thriller	Focus on the psychological and emotional tension that arises when characters are pushed to their limits, often involving themes of obsession, identity, and memory.	"Memento"
Mind bending	Complex, nonlinear, and surreal storytelling that challenges the audience's understanding of the story and leaves them questioning what is real and what is not.	"Inception"
Social criticism	Depiction of social and political issues, and the impact they have on individuals and society, often with a focus on the marginalized and oppressed.	"Parasite"
Metafictional	Depiction of characters who are aware they are in a film or fictional story, often playing with the audience's expectations and breaking the fourth wall.	"The Matrix"
black comedy	Use of humor and irony to criticize and mock contemporary social and political issues, often involving themes of death, violence, and the absurdity of life.	"Deadpool"
crime drama	Focus on the criminal underworld and the individuals who inhabit it, often involving themes of power, corruption, and morality.	"The Godfather"

[Xsanf]

As for TensorFlow.NET. I looked at your link. Yes, I have seen this project before. As is https://github.com/SciSharp/Torch.NET. In my opinion, with all the positive undertakings, it will be quite difficult for me personally to move from its current state to the existing LLM models)) It's a pity I'm not 17 years old))

That's why I was so happy with your work that this is so far the only project (which I found) that allowed me to run the model under Unity, literally in a couple of fixes. Everything else requires orders of magnitude more work. I'm not lazy)) I'm just not so productive anymore))

[Xsanf]

I'm sorry. There really is a problem. I understand now. The model that is loaded via Dll is really not understood in build as a resource. Now I'm figuring out how to do it right. Everything will work in the Unity editor, but in build the model is lost and cannot be loaded. I do not consider crutches when you can simply initially specify the path to the model and then simply place it in the right place. Creating addressable loadable resources in Unity is a rather confusing topic)) Since loading optimization is very important for games, the loading itself is due to a number of requirements. I'll change the example as soon as I figure it out. Such assets should be located in a special StreamingAssets directory. I'm trying to figure out the example of another asset for Unity. https://github.com/Babilinski/vosk-stt-unity I think it may be interesting for you too, and not only as an example. This is a very well-functioning standalone speech recognition system. I have already used it in Unity. It works with 18 languages and dialects - English, Indian English, German, French, Spanish, Portuguese, Chinese, Russian, Turkish, Vietnamese, Italian, Dutch, Catalan, Arabic, Greek, Farsi, Filipino, Ukrainian.

[Xsanf]

I changed LLaMa_Unity.unitypackage in my repository. The problem was that Unity included in the project files linked by dependencies, or not linked, obtained from the file system or from a network source, but only from the Assets/StreamingAssets directory. There are a lot of possibilities, download from archive, asynchronous download or download from server. But I just place the model and the text file with the role in this directory. Now the example is both launched in the editor and build is done. Just don't forget to download and place the model in the StreamingAssets directory.

[Xsanf]

I updated my repository to include the entire project. It's cleaner and more complete than the package, plus it already includes settings. This is in case there are any problems with launching from the package. In addition, I removed the extra dll, since I transferred them purely formally and did not immediately realize that the second one was just a copy with a different name that NativeApi.cs was not looking for)))

[martindevans]

It looks like there was a lot of discussion here. I haven't read all of it, so apologies if I missed something important!

Going back to the original issue. LLamaSharp 0.7.0 included some major fixes for text decoding in the StatelessExecutor (only stateless, the others still need more work). This should fix all complex characters which are encoded as multiple tokens (e.g. test case was multi byte korean characters and emoticons).