search

bigcode-project / starcoder.cpp

C++ implementation for 💫StarCoder
445 stars 36 forks source link

context memory pool #3

Closed Leterax closed 1 year ago

Leterax commented 1 year ago

When running the StarCoder model quantized using "q5_1" with a medium-sized context (3500 tokens) I run into this error:

ggml_new_tensor_impl: not enough space in the context's memory pool (needed 412241472, available 411791504)
Segmentation fault (core dumped)

How can I increase the context memory pool size?

NouamaneTazi commented 1 year ago

Yes, I'm able to reproduce this issue! We're investigating a fix here: https://github.com/ggerganov/ggml/pull/175