Some generation issues.

[cynthia-nlp]

I encountered some problems when using Llama2-70b-chat to generate some sentences. Specifically, I constructed a prompt template similar to:

prompt = 'Please write some sentences for sentiment classification. Here are some examples: ' \
         '[Demonstrations]. ' \
         'Please write more similar sentences.'

The corresponding code is implemented as:

prompt = 'Please generate some sentences for sentiment classification. Here are some examples: \n\n'
demos = random.sample(sentences, k=5)
random.shuffle(demos)
for d in demos:
    prompt += f"{d}\n"
prompt += f"\nPlease write more similar sentences."

dialogs: List[Dialog] = [
    [{"role": "user", "content": prompt}]
]
results = generator.chat_completion(
    dialogs,  # type: ignore
    max_gen_len=max_gen_len,
    temperature=temperature,
    top_p=top_p,
)

sentences is a list of strings from which I randomly sample five sentences as demonstrations. After running, the output of Llama either does not answer the question, or it freezes and does not respond. However, if I modify the code to:

dialogs: List[Dialog] = [
    [{"role": "user", "content": """
Please generate some sentences for sentiment classification. Here are some examples: \n\n
I felt ecstatic when I received the good news about my promotion at work.\n
She was heartbroken when she found out her best friend had betrayed her.\n
After a long day of hard work, I feel content and satisfied with what I have accomplished.\n
The movie was so touching that it brought tears to my eyes.\n
He was filled with anger and frustration when he realized he had been lied to.\n
\nPlease write more similar sentences.
    """}]
]
results = generator.chat_completion(
    dialogs,  # type: ignore
    max_gen_len=max_gen_len,
    temperature=temperature,
    top_p=top_p,
)
# print(prompt, '\n')
for dialog, result in zip(dialogs, results):
    for msg in dialog:
        print(f"{msg['role'].capitalize()}: {msg['content']}\n")
    print(f"> {result['generation']['role'].capitalize()}: {result['generation']['content']}")
    print("\n==================================\n")

The code runs successfully. I tried commenting out different parts and found that the code runs successfully when I remove the following:

for d in demos:
    prompt += f"{d}\n"

So what went wrong, and why does string concatenation cause decoding to fail?

[BabaWolo]

The issue you encounter is caused by the "+=" operator. "+=" is used for concatenation, meaning you're trying to concatenate the string "f"{d}\n" to the array variable prompt. Try using prompt.append("f"{d}\n") instead and see if that solves the issue.

[cynthia-nlp]

The issue you encounter is caused by the "+=" operator. "+=" is used for concatenation, meaning you're trying to concatenate the string "f"{d}\n" to the array variable prompt. Try using prompt.append("f"{d}\n") instead and see if that solves the issue.

Thank you for your reply! prompt is a string variable and I verified this with isinstance(). After concatenating, prompt is still a string variable. When I set max_seq_length to 512, I found that it got stuck when decoding to about 500 during generation. Later, by coincidence, I set a random seed and the problem was solved, although I still don't know why.