Setting up a gaming bot and getting to know itself

[JekSun97]

I once advised you to add GPT4ALL, since there is a function with which you could set the nature of the answers, what the model likes, who she is, etc. For example, I tried to write something like “you’re an old magician’s assistant, you like books, you answer angrily,” and the bot really answered like that (even insulted me somewhere), when you ask him what he likes, he answered something like this: “I love books, but you This isn't relevant, so leave me alone."

Is there some kind of function so that you can set the character of the bot? or does llama not support this?

[JanWerder]

Yes, that's what the instruct models are really good at. Please check this example by meta: https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/

[JekSun97]

Yes, that's what the instruct models are really good at. Please check this example by meta: https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/

Wow!, this is nice, but it looks like this plugin doesn’t have such a function?

[JanWerder]

It's not a function per-se but part of how you design the prompt. The model has been instructed to react to these keywords and function accordingly. godot-llm even has properties and functions to support this prompting, but I believe there is a bug right now. (https://github.com/Adriankhl/godot-llm/issues/8) But the basic version of just designing your prompt this way works.

[JekSun97]

It's not a function per-se but part of how you design the prompt. The model has been instructed to react to these keywords and function accordingly. godot-llm even has properties and functions to support this prompting, but I believe there is a bug right now. (#8) But the basic version of just designing your prompt this way works.

can you write a code example? Which function should I write in and what should my role look like in it, and the role of the model with the user’s question, thanks in advance!

[JanWerder]

Everything you need is already in the answers. If this isn't enough for you, check the sample in the provided issue I linked. That should help you.

[JekSun97]

Everything you need is already in the answers. If this isn't enough for you, check the sample in the provided issue I linked. That should help you.

on the site that you gave me, there is a completely different json syntax, unlike what is written in this plugin in the README, I am not a professional in llama, I want to clarify this.

I understood a little the meaning of this syntax: var _person_schema = { "type": "object", "properties": { "name": { "type": "string", "minLength": 3, "maxLength": 20, }, "birthday": { "type": "string", "format": "date" }, "weapon": { "enum": ["sword", "bow", "wand"], }, "description": { "type": "string", "minLength": 10, }, }, "required": ["name", "birthday", "weapon", "description"] }

But there are many questions: what types of “type” parameters are there? Are these parameters built into the model or can you invent and create them yourself? How can I set ready-made parameters? I searched all over Google to find some information about json syntax in llama

[Adriankhl]

Everything you need is already in the answers. If this isn't enough for you, check the sample in the provided issue I linked. That should help you.

on the site that you gave me, there is a completely different json syntax, unlike what is written in this plugin in the README, I am not a professional in llama, I want to clarify this.

I understood a little the meaning of this syntax: var _person_schema = { "type": "object", "properties": { "name": { "type": "string", "minLength": 3, "maxLength": 20, }, "birthday": { "type": "string", "format": "date" }, "weapon": { "enum": ["sword", "bow", "wand"], }, "description": { "type": "string", "minLength": 10, }, }, "required": ["name", "birthday", "weapon", "description"] }

But there are many questions: what types of “type” parameters are there? Are these parameters built into the model or can you invent and create them yourself? How can I set ready-made parameters? I searched all over Google to find some information about json syntax in llama

Let me explain it a bit more. This is not about json syntax, it is about how you design your input prompt. Instead of writing generate_text_simple("What can you help me with?"), write something like

generate_text_simple("<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful AI assistant for travel tips and recommendations<|eot_id|><|start_header_id|>user<|end_header_id|>

What can you help me with?<|eot_id|><|start_header_id|>assistant<|end_header_id|>")

This is model specific and the above prompt only works for llama3 instruct. It requires you to check the documentation to set the correct prompt, prefix and suffix. In the future I may automate this for popular models, but this is no my top priority for now.

[felix11111]

Hi, in the first instance I really appreciate that you are creating and maintaining this plugin for Godot. For a project I have, I'm creating a chat memory for the text LLM, as I noticed it doesn't remember the previous questions. And the AI personality issue, I placed it as context on the output, as I have yet to see a way to infuse context to the model outside of the output, I show you a bit of my code so you have some context:

extends Node

var total_time = 0.9
var default_prompt = “hello, what's your name”.
var conversation_history = “”
var personalization_AI = “<|assistant|>My name is B59, I am an advanced artificial intelligence created by Vault-Tec in the days before the great cataclysm. <|end|>”
var first_conversation = true
var prompt = “”

# wizard parameters

func _ready():
    $GDLlama.n_predict = 200
    $entrada.text = default_prompt
    $GDLlama.reverse_prompt = “.”

func _process(delta):
    total_time += delta
    if total_time > 0:
        if $GDLlama.is_running():
            call_active()
        else:
            llama_inactive()
        total_time = 0.0

func llama_active():
    $Enter.disabled = true

llama_inactive() func:
    $Enter.disabled = false

func _on_enter_pressed():
    var user_input = $entry.text
    if user_input != “”:
        if first_conversation:
            prompt = customization_AI + “user_user: ‘ + user_input + ’\n” + “\nAI:”
            first_conversation = false
        else:
            prompt = conversation_history + “\nUser: ‘ + user_input + ’\n” + “\nAI:” 

    $GDLlama.run_generate_text(prompt, “”, “”) 

func _on_gd_llama_generate_text_updated(response: String):
    $output.text += response 
    conversation_history = response 

Do not add conversation_history += answer as it duplicates the conversation for me. I don't have much practice in the whole scope of how to set up a Model, I am using the Phi-3-mini-128k-instruct model. Sorry for my lack of information, I've been trying to solve this problem for a couple of days now.

[JanWerder]

It depends on the flags you set in the plugin, but what you need to do is save the finished to your conversation_history. This could be either when the generate_text_finished event is triggered or when the input_wait_started comes up. (See https://github.com/Adriankhl/godot-llm/issues/9 for the difference). When your conversation_history then contains the complete conversation + proper tokens for the AI to do the next turn, it should remember its persona and answer the way you would like it to.

PS. The prompt format for phi-3 is different from llama, but should work like this:

<|user|>
I am going to Paris, what should I see?<|end|>
<|assistant|>
Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n\n1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n\nThese are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world."<|end|>
<|user|>
What is so great about #1?<|end|>
<|assistant|>

[Adriankhl]

@felix11111 This is an interesting topic, so I have spent some time to prepare a non-stop chat example between 2 AI.

extends Node

class ChatData:
    extends RefCounted
    var character: String
    var text: String

var personal_prompt_a = "Your name is B59, you are an advanced artificial intelligence created by Vault-Tec in the days before the great cataclysm."
var conversation_history: Array[ChatData] = []
var current_character = ""
var max_history = 2

func _ready():
    $GDLlama.context_size = 4096
    $GDLlama.n_predict = 1024
    $GDLlama.should_output_prompt = false
    $GDLlama.should_output_special = false

func _process(_delta):
    if ($GDLlama.is_running()):
        $StartButton.disabled = true;
    else:
        $StartButton.disabled = false

func prompt_with_history() -> String:
    if (conversation_history.size() > max_history):
        conversation_history.remove_at(0)
    var prompt = ""
    if (current_character == "a"):
        for d in conversation_history:
            if (d.character == "b"):
                prompt += "<|user|>\n" + d.text + "<|end|>\n"
            else:
                prompt += "<|assistant|>\n" + d.text + "<|end|>\n"
    else:
        for d in conversation_history:
            if (d.character == "a"):
                prompt += "<|user|>\n" + d.text + "<|end|>\n"
            else:
                prompt += "<|assistant|>\n" + d.text + "<|end|>\n"
    prompt += "<|assistant|>\n"

    #print("Current character: " + current_character)
    #print(prompt)

    return prompt

func _on_start_button_pressed():
    current_character = "a"
    var prompt: String =  "<|user|>\n" + personal_prompt_a + "<|end|>\n" + "<|assistant|>\n"

    $GDLlama.run_generate_text(prompt, "", "")

func _on_gd_llama_generate_text_finished(text):
    print(current_character + ":")
    print(text)
    var data = ChatData.new()
    data.character = current_character
    data.text = text
    conversation_history.append(data)
    if (current_character == "a"):
        current_character = "b"
    else:
        current_character = "a"
    var prompt: String = prompt_with_history()
    $GDLlama.run_generate_text(prompt, "", "")

I have tested it with a few different phi3-mini gguf model, and I observe something strange. The official model uses a older conversion script, so there is a <|end|> at the end of generated text even if should_output_special is disabled. However, the generated text is much better than those without <|end|>, as if there is a bug in those gguf. Definitely keep an eye on the choice of gguf if you are using phi3-mini.

There are a few subtleties if you want to keep the chat history -

1. Yes, LLM models don't keep the history (in technical term, stateless), the "memory" comes from either (1) you store the output and use it in the next input or (2) you run it in an interactive mode so the text generation is not done yet.
2. The Context Size of a model is limited (also limited by your machine hardware), you can't just put infinite length of chat history there. In my example, I set $GDLlama.context_size = 4096, $GDLlama.n_predict = 1024 and only keep the latest 2 chats to ensure it doesn't exceed the chat limit, you can improve it by actually detecting the size of the history and set the parameters dynamically.
3. There can be important information in the beginning of the conversation, so you may want to keep the beginning and the end instead, though you need to carefully design and test the prompt there. In fact, the interactive mode has N Keep for this.
4. To go a bit further, sometimes, important information is in the middle of the conversation. Here comes the LlmDB node, which allows you to store text to a database alongside with a vector to represent the semantic meaning of the text, and you can retrieve the text from the database by getting the most similar texts (assume similar means important) to your prompt. This requires further adjustment on how you design your prompt. I am planning to add a demo in my LLM template about this based on the above infinite chat example 😄