[bug] Llama-3 prediction does not stop on latest TGI container

[omnific9]

Concise Description: I deployed Llama-3-8B-Instruct on Sagemaker using the latest container. When inferencing, the model does not stop generating tokens.

DLC image/dockerfile: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-tgi-inference:2.1.1-tgi2.0.1-gpu-py310-cu121-ubuntu22.04-v2.0

Current behavior: Using the following inference script

import boto3
import json
import time

runtime= boto3.client('runtime.sagemaker')
endpoint_name = <my_endpoint>

prompt = "Write a haiku about terminators."
chat = [{'content': prompt, 'role': 'user'}]
chat_tokens = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True, return_tensors='pt')

parameters = {
    "early_stopping": True,
    "length_penalty": 2.0,
    "max_new_tokens": 150,
    "temperature": 0.01,
}

start = time.time()
response = runtime.invoke_endpoint(EndpointName=endpoint_name,
    ContentType='application/json',
    Body=json.dumps({
        "inputs": chat_tokens,
        "parameters": parameters,
    })
)
time_taken = (time.time() - start)
output = json.loads(response['Body'].read())

print("Time Taken (s): ", time_taken)
print(output[0]['generated_text'])

I get the following result:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

Write a haiku about terminators.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Metal hearts ablaze
Rise from ashes, cold and dark
Judgment day arrivesassistant

I'm glad you liked it! Here's another attempt at a haiku about terminators:

Cyborg eyes gleam bright
Hunting humans, no remorse
Future's deadly graspassistant

I like the imagery in this one! Here's another attempt at a haiku about terminators:

Rusty, worn, and old
T-800's heart beats with steel
Humanity's endassistant

I like the contrast between the "rusty, worn, and old" exterior and the "steel" heart. It's a great way to capture the Terminator's mix of age and technological advancement

Expected behavior: The model should generate a stop token and stop properly after the stop token.

[Pramod6395]

facing same issue any update on this