search

Stability-AI / StableLM

StableLM: Stability AI Language Models
Apache License 2.0
15.85k stars 1.04k forks source link

StableVicuna does not stop dialog speaking, probably until max_new_tokens. #70

Closed ycat3 closed 1 year ago

ycat3 commented 1 year ago

Now, StableVicuna is working on my local PC(ubuntu22.04). However, unexpected another input and answer are following until max_tokens, perhaps. Is it normal ? Or missing special token ? See more detail info here.

huzama commented 1 year ago

The root cause of this issue is using "vicuna-13b-delta-v0" as the base model for stable-vicuna, which has a problem with the tokenizer. The issue can be resolved by using "vicuna-13b-delta-v1.1" instead. I guess we need to wait for stable_vicuna_v1.1 to be released before the problem can be fully resolved.

ycat3 commented 1 year ago

Thank you for your advice. Vicuna-13b-v1 also had same kind of problem(tokenizer special token) and was fixed soon. Now vicuna-13b-v1.1 is working fine at low cost GPU environment. See here. We might have to wait StableVicuna-v1.1.

mcmonkey4eva commented 1 year ago

so, first: this is the repo for StableLM, not StableVicuna. Second: this issue isn't a Vicuna thing, this is how all LLMs operate. They just predict what comes next. It is down to the software running it to split it into Q&A format for Vicuna usage. So, you have to set a stop string in whatever software you're using. Normally the stop string is \n### Human: or \n### or similar. If in doubt, you can use software like https://github.com/oobabooga/text-generation-webui which supports vicuna properly out-of-the-box.