Open pedro-devv opened 8 months ago
I didn't notice the panic on my end, using both streaming and non streaming methods.
I said panic, but that's not exactly correct, what happens is an error in llama.cpp
's side, I forget if it's an exception or something else. Either way, this depends on how large you make your context/session, if you make it large enough and never use the same session for too long, you probably won't ever notice it, but this definitely still happens.
I see, I was using ~2048 context length so I didn't notice at all
At the moment when the max context size is reached, the program panics. Methods such as sliding context window should be implemented.