Adding End-Of-Generation-Token parameter for text generation Inference API

[aryananumula]

Is your feature request related to a problem? Please describe. While using the Inference API for a chatbot-style text-generation model, such as openchat-3.5, it is not possible to set an end of generation token.

Describe the solution you'd like Addition of the end_of_generation_token parameter to the Inference API for text generation models.

Describe alternatives you've considered Setting max_new_tokens to 1, and then generating new tokens and looking for a certain token to stop at.

Additional context There is no additional context for this request.

[MichaelVandi]

You can do something like this

{
    "inputs": "What is Deep Learning?",
    "parameters": {
        "max_new_tokens": 300,
        "stop": ["<|end_of_text|>", "<|endoftext|>", "}"]
    }
}

Where parameters.stop is an array of eos tokens