why the transformers vicuna-7b-v1.1 results is different from vllm vicuna-7b-v1.1 results, How to solve it，thanks

[lijianxing123]

transformers vicuna-7b-v1.1 script:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("lmsys/vicuna-7b-v1.1")
model = AutoModelForCausalLM.from_pretrained("lmsys/vicuna-7b-v1.1").to(0)

import time
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(0)
    generate_ids = model.generate(inputs.input_ids, max_length=256)
    result = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
    print(result)

the results is:

Hello, my name is Dr. David C. Preston, and I am a chiropractor serving the residents of the greater Phoenix area. I have been in practice for over 20 years, and I am dedicated to helping my patients achieve optimal health and wellness through natural, non-invasive chiropractic care.
At my practice, I offer a wide range of chiropractic services, including adjustments, massage therapy, and rehabilitation. I also offer nutritional counseling and weight loss coaching to help my patients achieve their health goals.
I believe that every person deserves to live a healthy, happy life, and I am committed to helping my patients achieve this through chiropractic care. If you are looking for a chiropractor who will take the time to understand your unique needs and provide personalized care, I would be honored to help.
Please feel free to contact my office to schedule an appointment or to learn more about the services I offer. I look forward to helping you achieve optimal health and wellness.

The president of the United States is a man who has been accused of sexual assault by more than a dozen women, and who has bragged about committing sexual assault. He is a man who has been accused of rape by at least one woman, and who has been accused of paying hush money to cover up his sexual misconduct. He is a man who has been accused of sexual harassment by at least one woman, and who has been accused of using his position of power to force himself on women. He is a man who has been accused of sexual battery, and who has been accused of engaging in other forms of sexual misconduct. He is a man who has been accused of sexual assault by multiple women, and who has been accused of using his position of power to force himself on women. He is a man who has been accused of sexual misconduct by multiple women, and who has been accused of using his position of power to force himself on women. He is a man who has been accused of sexual misconduct by multiple women, and who has been accused of using his position of power to force himself on women. He is a man who has been accused of sexual misconduct by multiple women, and

The capital of France is Paris.
The currency of France is the Euro.
The official language of France is French.
The population of France is approximately 67 million people.
The flag of France is a tricolor flag with three horizontal bands of blue, white, and red.
The national symbol of France is the Gallic rooster.
The national sport of France is football (soccer).
The national animal of France is the Greyhound.
The national flower of France is the Rose.
The national tree of France is the Oak tree.
The national bird of France is the Hoopoe.
The national monument of France is the Eiffel Tower.
The national dish of France is Coq au Vin.
The national drink of France is wine.
The national holiday of France is Bastille Day, which is celebrated on July 14th.
The national anthem of France is "La Marseillaise".
The highest mountain in France is Mont Blanc.
The largest city in France is Paris.
The smallest country in the world is Vatican City, which is located within Rome, Italy.
The largest island in the world is Greenland.

The future of AI is bright, and it’s up to us to ensure that it benefits everyone. As we continue to develop and deploy AI technologies, we must prioritize ethical considerations and work to mitigate potential negative impacts. By doing so, we can create a future where AI is a force for good, driving progress and improving lives around the world.

[lijianxing123]

vllm vicuna-7b-v1.1 script:

from vllm import LLM, SamplingParams
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

sampling_params = SamplingParams(max_tokens=256)

# Create an LLM.
llm = LLM(model="lmsys/vicuna-7b-v1.1") 
outputs = llm.generate(prompts, sampling_params)
# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

the results is:

Prompt: 'The future of AI is', Generated text: 'here, and it’s called “Generative AI”. In this podcast, we introduce you to the key players in the industry and what makes Generative AI so special:</s>'

Prompt: 'The president of the United States is', Generated text: 'the leader of the free world, responsible for guiding the nation and its policies. In the past, no other occupation of an American president has been confirmed from flying a plane to being the mayor of a small town. However, being president of the United States is an important job requirement. It requires an individual who is willing to take on the challenges of office and lead the country through both good times and bad.</s>'

Prompt: 'Hello, my name is', Generated text: "D'Arcy, and I' Yeah, I know, it's a bit of a mouthful. My parents had high hopes for me, and then they met my sister. Anyway, I've been doing this stand-up comedy thing for a couple of years now, and I've been lucky enough to perform all over Scotland. My comedy is a bit observational, and I like to poke fun at myself and the world around us. I hope you enjoy it!</s>"

Prompt: 'The capital of France is', Generated text: 'Paris, which is actually split into two main districts. The first is the City of Paris, which is the most central part of the city, and it is home to iconic landmarks such as the Eiffel Tower, the Louvre Museum, Notre Dame Cathedral, and the Pompidou Center. The second is La banlieue, which refers to the outer suburbs of Paris. This mainly residential area covers around 1,500 square kilometers, and it is home to around four million people. Many of La banlieue residents are immigrants or children of immigrants, who have moved to Paris looking for work or a better life. While the suburbs are considered somewhat isolated from the rest of France, they have a rich culture of their own, with many unique neighborhoods and a diverse mix of people.\n\nImportant landmarks in La banlieue include various sports stadiums, such as the Stade de France, the largest stadium in France, which is located in Saint-Denis, and the Parc des Princes stadium, which is home to the Paris Saint-Germain football club. There are also several theme parks in La banlieue, such'

[normster]

The default sampling parameters for vllm and the default generation parameters for huggingface seem to differ. For one, huggingface uses greedy decoding by default while vllm uses sampling. Does vllm still differ when set to greedy decoding?

[zhuohan123]

Close this issue since it seems like to be the incorrect setting of sampling parameters. Feel free to reopen the issue if you meet any further problems!