EM_German/examples/EM_German_7b_Mistral_GGUF_Demo.ipynb : Hallucinations

[te0006]

Hello Jan, very good work!

I experimented with the Structured Output mode in EM_German_7b_Mistral_GGUF_Demo.ipynb. The designated attribute slots are indeed correctly filled quite reliably if relevant info is present in the user input.

However, if e.g. very short and content-free user inputs are supplied, the model often fills several attrributes nonetheless, i.e. hallucinates. (Even though these attributes were not desiganted as "required".) Example below.

Any idea on how to reduce or even eliminate this problem? That, is apart from (brittle) prompt engineering attempts, i.e. to politely ask the model to be more strict, to not output info that wasn't present in the input, or similar.

schema={
  'type': 'object',
  'properties': {
    'name': {'type': 'string'},
    'ort': {'type': 'string'},
    'jahr': {'type': 'integer'}},
  'required': ['name', 'ort', 'jahr'],
  required': [],
}
...
instruction="Max schläft."
output=llm(
    f"Bitte gebe gültiges JSON aus. USER: {instruction} ASSISTANT:",
    max_tokens=200,
    temperature=0.3,
    grammar=grammar)
json.loads(output['choices'][0]['text'])

-->  {'jahr': 2018, 'name': 'Max', 'ort': 'Hamburg'}

[jphme]

Hey, I just did a few small experiment and also didn't find a good ad-hoc solution for this.

You'd probably have to give the model some kind of "escape" mechanism to prevent hallucinations or use some kind of enum or bool to verify presence of the required information in the given text. TBH I am also not very familiar with the GBNF Grammar.

If this is a common usecase I will probably add a few more examples with JSON Output (and optional/fields) to the finetuning data for the next model generation...

[jphme]

Closing as there is no immediate fix. We will add some better training data for this for the next generation, if you have any questions (or suggestions/use cases) in the meantime, I am happy to chat.

[te0006]

We will add some better training data for this for the next generation

Im skeptical on whether this can help.

The model is forced to generate to generate output conforming to a grammar, but if the grammar permits arbitrary-length (syntactically conformant of course) sequences, like a JSON grammar does and should do, the generation process is never terminated "by the grammar". Instead, the model always continues to generate tokens as it pleases (well until some stop criteria in the decoder is met). This often seems to result in the filling of additional JSON object attributes (i.e. key/value pairs) for which no corresponding information whatsoever was present in the user utterance - in other words, attributes are "hallucinated". Perhaps there is a way to persuade the model with clever prompting to only peruse information that actally was present in the user utterance,