Keyname "name" GGUF saving

[marscuspolos]

I have been trying to save the GGUF after finetunning this notebook. I don't change any settings and keep getting the same error KeyError - "name" and with Qwen 2.5 i got KeyError - "type". I didn't test Qwen recently, last time was saturday. Any suggestion on how to fix this problem or a workaround? Llama 3.2 Conversation

[danielhanchen]

Yep can reproduce - will fix - sorry about this!

[cool9203]

Can reference this solution

This is my test code, is work. Won't get KeyError when format ollama modelfile data.

# coding: utf-8

from __future__ import print_function

import string

llama31_ollama = '''
FROM {__FILE_LOCATION__}
TEMPLATE """{{ if .Messages }}
{{- if or .System .Tools }}<|start_header_id|>system<|end_header_id|>
{{- if .System }}

{{ .System }}
{{- end }}
{{- if .Tools }}

You are a helpful assistant with tool calling capabilities. When you receive a tool call response, use the output to format an answer to the orginal use question.
{{- end }}
{{- end }}<|eot_id|>
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}
{{- if eq .Role "user" }}<|start_header_id|>user<|end_header_id|>
{{- if and $.Tools $last }}

Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.

Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.

{{ $.Tools }}
{{- end }}

{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- else if eq .Role "assistant" }}<|start_header_id|>assistant<|end_header_id|>
{{- if .ToolCalls }}

{{- range .ToolCalls }}{"name": "{{ .Function.Name }}", "parameters": {{ .Function.Arguments }}}{{ end }}
{{- else }}

{{ .Content }}{{ if not $last }}<|eot_id|>{{ end }}
{{- end }}
{{- else if eq .Role "tool" }}<|start_header_id|>ipython<|end_header_id|>

{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- end }}
{{- end }}
{{- else }}
{{- if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}{{ .Response }}{{ if .Response }}<|eot_id|>{{ end }}"""
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER stop "<|eom_id|>"
PARAMETER temperature 1.5
PARAMETER min_p 0.1
'''

class SafeFormatter(string.Formatter):
    """Reference: https://stackoverflow.com/a/34033230"""

    def vformat(self, format_string, args, kwargs):
        args_len = len(args)  # for checking IndexError
        tokens = []
        for lit, name, spec, conv in self.parse(format_string):
            # re-escape braces that parse() unescaped
            lit = lit.replace("{", "{{").replace("}", "}}")
            # only lit is non-None at the end of the string
            if name is None:
                tokens.append(lit)
            else:
                # but conv and spec are None if unused
                conv = "!" + conv if conv else ""
                spec = ":" + spec if spec else ""
                # name includes indexing ([blah]) and attributes (.blah)
                # so get just the first part
                fp = name.split("[")[0].split(".")[0]
                # treat as normal if fp is empty (an implicit
                # positional arg), a digit (an explicit positional
                # arg) or if it is in kwargs
                if not fp or fp.isdigit() or fp in kwargs:
                    tokens.extend([lit, "{", name, conv, spec, "}"])
                # otherwise escape the braces
                else:
                    tokens.extend([lit, "{{", name, conv, spec, "}}"])
        format_string = "".join(tokens)  # put the string back together
        # finally call the default formatter
        return string.Formatter.vformat(self, format_string, args, kwargs)

llama31_ollama = llama31_ollama.replace("{{", "⚫@✅#🦥").replace("}}", "⚡@🦥#⛵")

llama31_ollama = SafeFormatter().format(llama31_ollama, __FILE_LOCATION__="===location===")

llama31_ollama = llama31_ollama.replace("⚫@✅#🦥", "{{").replace("⚡@🦥#⛵", "}}").rstrip()

print(llama31_ollama)

Output:

FROM ===location===
TEMPLATE """{{ if .Messages }}
{{- if or .System .Tools }}<|start_header_id|>system<|end_header_id|>
{{- if .System }}

{{ .System }}
{{- end }}
{{- if .Tools }}

You are a helpful assistant with tool calling capabilities. When you receive a tool call response, use the output to format an answer to the orginal use question.
{{- end }}
{{- end }}<|eot_id|>
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}
{{- if eq .Role "user" }}<|start_header_id|>user<|end_header_id|>
{{- if and $.Tools $last }}

Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.

Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.

{{ $.Tools }}
{{- end }}

{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- else if eq .Role "assistant" }}<|start_header_id|>assistant<|end_header_id|>
{{- if .ToolCalls }}

{{- range .ToolCalls }}{"name": "{{ .Function.Name }}", "parameters": {{ .Function.Arguments }}}{{ end }}
{{- else }}

{{ .Content }}{{ if not $last }}<|eot_id|>{{ end }}
{{- end }}
{{- else if eq .Role "tool" }}<|start_header_id|>ipython<|end_header_id|>

{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- end }}
{{- end }}
{{- else }}
{{- if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}{{ .Response }}{{ if .Response }}<|eot_id|>{{ end }}"""
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER stop "<|eom_id|>"
PARAMETER temperature 1.5
PARAMETER min_p 0.1

Quick apply this solution:

# coding: utf-8

from __future__ import print_function

import string
from unittest.mock import patch

import unsloth.save

class SafeFormatter(string.Formatter):
    """Reference: https://stackoverflow.com/a/34033230"""

    def vformat(self, format_string, args, kwargs):
        args_len = len(args)  # for checking IndexError
        tokens = []
        for lit, name, spec, conv in self.parse(format_string):
            # re-escape braces that parse() unescaped
            lit = lit.replace("{", "{{").replace("}", "}}")
            # only lit is non-None at the end of the string
            if name is None:
                tokens.append(lit)
            else:
                # but conv and spec are None if unused
                conv = "!" + conv if conv else ""
                spec = ":" + spec if spec else ""
                # name includes indexing ([blah]) and attributes (.blah)
                # so get just the first part
                fp = name.split("[")[0].split(".")[0]
                # treat as normal if fp is empty (an implicit
                # positional arg), a digit (an explicit positional
                # arg) or if it is in kwargs
                if not fp or fp.isdigit() or fp in kwargs:
                    tokens.extend([lit, "{", name, conv, spec, "}"])
                # otherwise escape the braces
                else:
                    tokens.extend([lit, "{{", name, conv, spec, "}}"])
        format_string = "".join(tokens)  # put the string back together
        # finally call the default formatter
        return string.Formatter.vformat(self, format_string, args, kwargs)

def create_ollama_modelfile(tokenizer, gguf_location):
    """
    Creates an Ollama Modelfile.
    Use ollama.create(model = "new_ollama_model", modelfile = modelfile)
    """
    modelfile = getattr(tokenizer, "_ollama_modelfile", None)
    if modelfile is None:
        return None

    modelfile = modelfile.replace("{{", "⚫@✅#🦥").replace("}}", "⚡@🦥#⛵")

    if "__EOS_TOKEN__" in modelfile:
        modelfile = SafeFormatter().format(
            modelfile,
            __FILE_LOCATION__=gguf_location,
            __EOS_TOKEN__=tokenizer.eos_token,
        )
    else:
        modelfile = SafeFormatter().format(
            modelfile,
            __FILE_LOCATION__=gguf_location,
        )

    modelfile = modelfile.replace("⚫@✅#🦥", "{{").replace("⚡@🦥#⛵", "}}").rstrip()

    return modelfile

# Method 1
unsloth.save.create_ollama_modelfile = create_ollama_modelfile

# Method 2
@patch.object(unsloth.save, "create_ollama_modelfile", create_ollama_modelfile)
def train_model(...)
    ...

[danielhanchen]

@marscuspolos @cool9203 Apologies on the delay - just fixed this! If you're on Colab or Kaggle, delete and disconnect then restart). If you installed Unsloth on a local machine, please update it via:

pip uninstall unsloth -y
pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

Thanks @cool9203 for the proposed solution - I instead took a more "direct dumber" approach and just replace all "{", "}" with some symbols!

[marscuspolos]

Hello. Thank you for fixing it. I tested the notebook with my data and worked without problem.

Amazing work

[shimmyshimmer]

Hello. Thank you for fixing it. I tested the notebook with my data and worked without problem.

Amazing work

Amazing glad it worked! closing this issue now :)