search

meta-llama / codellama

Inference code for CodeLlama models
Other
15.44k stars 1.79k forks source link

How to let codellama or codellama-python stop? #155

Open yangzhj53 opened 8 months ago

yangzhj53 commented 8 months ago

When I use codellama or codellama-python to finish the continuation of a prompt, a lot of '\n' are outputed in the end until it reaches the max_gen_len. Is there any way to let it stop early except limiting the max_gen_len which is not convenient?

yangzhj53 commented 8 months ago

I read the code and I found a trick to solve it. After encoding, the id of '\n' is 13. The program will generate a batch of token_id each time, and I just need to check if 13 appears in all positions for two times.

abs-xyz commented 8 months ago

Maybe use the stop_token in generate()?

humza-sami commented 7 months ago

Use </s> as end of string token in the generation