llama: significance of truncating input to `ctxLen - 4`

[iboB]

Why truncate to ctxLen - 4? Why is that 4 significant.

This is kept for now as per llama.cpp demos, but we should investigate.

[pminev]

I was looking at the commits and PRs:

• https://github.com/ggerganov/llama.cpp/issues/71
• https://github.com/ggerganov/llama.cpp/commit/e2d490dafd860eaaaf9aa8008ab790527d556daf
• this was a linked problem: https://github.com/ggerganov/llama.cpp/pull/1789/files,

Finally I talked with G.Gerganov and it was added to secure space (at least 4) in KV cache for the new generated tokens. I didn't ask him further why it's explicitly 4, but it seems like when new tokens are generated the input will be truncated, in order to have enough space again.

[iboB]

this can be closed now. There is a link to this issue in the code for reference