set_context and advance_context with same text, generate different completion output

[FlorianDevPhynix]

I found an inconsistency between set_context and advance_context. If I use advance_context (with truncate_context(0)) the generated output is different then when I use set_context instead. This is with the same Session and Seed!

I've taken a look at the code and learned that set_context uses advance_context itself, so now I'm more confused than before. I tested a little bit and found that the output is the same if I use truncate_context(0) before set_context.

I also verified that the context, before generating completion output, is the exact same. Same text and token count. So it must be this inconsistency that results in different output. Some of the output is similar, but somehow the output when using set_context seems a little less intelligent.

Prompt:

 <s>system\n' +
'You are a helpful, respectful and honest assistant.\n' +
"If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information\n" +
'</s><s>user\n' +
'Why is the banana crooked?\n' +
'</s><s>assistant\n

Output with truncate_context(0) and advance_context:

" The banana's crooked appearance is due to its natural growth process. Bananas grow on a plant called Musa, which is a type of herbaceous plant. The bananas grow in clusters, and each individual banana grows at its own pace, resulting in a variety of shapes and sizes. The crooked appearance is not a sign of any defect or problem, but rather a natural characteristic of the banana."

Output with set_context:

" The banana's crooked appearance is due to its natural growth process. Bananas grow on a plant called Musa, which is a type of herbaceous plant. The bananas grow in clusters, and each individual banana has its own unique shape and size. The crookedness is a result of the banana's natural growth pattern, and it is not an indication of any defect or problem."

These are very similar but still have minor differences even though I used the same seed for their Session. I also ran two for each and it stayed the same.

session.truncate_context(0)
session.advance_context(input)

has the same output as

session.truncate_context(0)
session.set_context(input)

Any idea why this might be happening? I really have no clue here.

[FlorianDevPhynix]

When I'm using a German user input this issue is even more noticeable.

Prompt:

' <s>system\n' +
'You are a helpful, respectful and honest assistant.\n' +
"If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information\n" +
'</s><s>user\n' +
'Warum ist die Banane krumm?\n' +
'</s><s>assistant\n'

Output with truncate_context(0) and advance_context:

' Die Banane ist nicht krumm, sondern hat eine spezielle Form, die sich durch die Wachstumsbedingungen und die Umgebung der Pflanze entwickelt hat. Die Bananenpflanze wächst in Gruppen, und die Früchte sind an die Bedingungen angepasst, in denen sie wachsen. Die Bananenpflanze ist eine Monokarp, das heißt, sie blüht nur einmal und bildet dann Früchte, die dann reifen und geerntet werden. Die Bananenpflanze ist eine Pflanze, die in tropischen und subtropischen Gebieten wächst und benötigt eine bestimmte Temperatur und Feuchtigkeit, um sich zu entwickeln und zu überleben.\n' +
'</s>'

Output with set_context:

' Die Banane ist nicht krumm, sondern hat eine spezielle Form, die sich durch die Wachstumsbedingungen und die Art der Pflanze entwickelt hat. Die Form der Banane ist angepasst an die Verbreitung der Pflanze und die Bedürfnisse der Tiere, die die Banane fressen.'

The set_context output is imprecise and less detailed. It also seems to be wrong about "the Banana is adapted to the distribution of the plant and the needs of the animals".

Such big differences in output with the same prompt and Seed.