Open getnamo opened 8 months ago
NB: context meaning full prompt history
Likely need some form of batched inference with different contexts: https://github.com/ggerganov/llama.cpp/tree/master/examples/batched-bench
NB:
NB: context meaning full prompt history