After fine-tuning, the model repeat the answer and not stop

[abdoelsayed2016]

After fine-tuning, I have encountered an unexpected behavior where the model repeatedly generates the same answer without stopping.

i tried different models of llama

decapoda-research/llama-7b-hf yahma/llama-7b-hf here are some examples

 Tell me about alpacas.

 1. Alpacas are from South America.
2. Alpacas are from South America.
3. Alpacas are from South America.
4. Alpacas are from South America.
5. Alpacas are from South America.
6. Alpacas are from South America.
7. Alpacas are from South America.
8. Alpacas are from South America.
9. Alpacas are from South America.
10. Alpacas are from South America.
11. Alpac

who played charlie bucket in the original charlie and the chocolate factory

\begin{itemize}
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\item Freddie Highmore
\

Also, adapter_model.bin size is 1kb

[aryanq123]

can someone assign me this task i wants to work upon it

and explain briefly

[Qubitium]

@abdoelsayed2016 You using an old model that is no longer recommended as it was convertes to hf using wrong tokenizer params. Check the repo readme for valid 7b hf models.

[abdoelsayed2016]

@diegomontoya I have already read the readme many times. I see that while finetuning the base model, isdecapoda-research/llama-7b-hf I have two questions. First, why does the model repeat the answer? The model doesn't know when it should stop

The second question is whether the adapter weight is 1KB only. Is that normal?

[abdoelsayed2016]

Also i tired huggyllama/llama-7b

[abdoelsayed2016]

I see a lot of issues talking about the size of the adapter I tried all solutions, but still, the adapter is the same size 1KB.

anyone has faced the same issue cuz training is costly

[asenmitrev]

I was able to solve this issue with this comment. Uninstall peft and install from github commit.

I was desperate with the repetitions for both models (yahma and decapoda-research) but I think the root cause is the empty adapter model.bin.

If you want to do a quick test without training, take the last checkpoint pytorch_model.bin and replace the adapter_model.bin with that, renaming it. This allowed me to test the model and the repetition was fixed.

[abdoelsayed2016]

@asenmitrev

Yesterday I did this solution but still adapter model.bin size 1KB

pip uninstall peft -y
pip install git+https://github.com/huggingface/peft.git@e536616888d51b453ed354a6f1e243fecb02ea08

i tried again with uninstall peft. Here is the output

[abdoelsayed2016]

I also tried this solution https://github.com/huggingface/peft/issues/286 https://github.com/tloen/alpaca-lora/issues/37#issuecomment-1473140882

The adapter size now is 7 GB

And model back to repeat the answers.

I also changed the checkpoint model name from pytorch_model.bin to the adapter_model.bin

same output

[asenmitrev]

I tried decapoda research and the answer is repeated. yahma is fine

[abdoelsayed2016]

@asenmitrev Yahma with which version of peft? Because i tired it same problem I got confused They should add description in readme or fixed requirements

[asenmitrev]

The one quoted in my first answer.

[SimpleConjugate]

You should try adding repetition_penalty keyword argument to generation config in the evaluate function. repetition_penalty >1 should do it. I finetuned a model and used repetition_penalty=2 to resolve the problem for myself.