Configurable EOS Token Support

[ManilShrestha]

Description This pull request introduces the integration of models with different EOS tokens than those used by the default ExllamaV2 tokenizer. This change addresses the need for flexibility in model configurations without the necessity of hardcoding EOS tokens directly into the codebase.

Key Changes

• Dynamic EOS Token Configuration: A new optional configuration parameter, eos_token, has been added to the config.ini file. This parameter allows different models to specify their own EOS tokens directly via the configuration file, thus enabling seamless integration of models with unique tokenization requirements.

• Default Behavior: If the eos_token key is not present in config.ini, the system will default to using tokenizer.eos_token_id. This ensures backward compatibility and default behavior that aligns with the ExllamaV2 tokenizer's specifications.

• Custom EOS Token Support: For models requiring a distinct EOS token, such as Llama3, users can now specify this by adding eos_token = under the appropriate section in the config.ini file. This entry should be followed by the specific EOS token value as needed.

Usage Users who need to configure a custom EOS token for a model should add or update the eos_token entry in the config.ini file under the relevant model section. For example:

[llama3-70B]
string = llama3-70B
repo = <path to model>
specrepo = <path to spec model>
max_context = <max context len>
total_context = <total context len>
eos_token = <|eot_id|>

p.s. This should also cover PR #17 , as it was worked from the branch pushed yesterday.

[edk208]

can you make eos_tokens a list of tokens... to make things easier, maybe they can just be a list of token_ids. also maybe you should always add the tokenizers eos to the list no matter what...

[ManilShrestha]

The list of token_ids would be represented in config.ini like:

[llama3-70B]
string = llama3-70B
repo = <path to model>
specrepo = <path to spec model>
max_context = <max context len>
total_context = <total context len>
eos_token_ids = 128001,128009

Instead of having the strings for eos_tokens

eos_tokens = <|end_of_text|>,<|eot_id|>