Closed the-crypt-keeper closed 1 year ago
I figured it out, there's a couple of things going on here
--top_k 0
is definitely a bad idea as it produces undefined behavior, --top_k 1
is likely what we want here
The second problem is that this model really needs a repeat_penalty
to .. not repeat itself!
I've forked upstream ggml and implemented it here: https://github.com/the-crypt-keeper/ggml/tree/starcoder_repeat_penalty
Heres what happens when I follow the instructions in the README:
I have the same issue with
example/starcoder
from the ggml repo, it just keeps on generating and wont stop until it hits the generation limit. Running the original model through transformers with the same prompt produces the same input token ids, but the outputs frustratingly diverge.