search

ludwig-ai / ludwig

Low-code framework for building custom LLMs, neural networks, and other AI models
http://ludwig.ai
Apache License 2.0
11.11k stars 1.19k forks source link

Ensure correct padding token for Phi and Pythia models #3899

Closed arnavgarg1 closed 8 months ago

arnavgarg1 commented 8 months ago

Follow up from https://github.com/ludwig-ai/ludwig/commit/9eb113b7b407ab5dd0e7c1f69f715fb8af8ca85e

Ensures that all of the tokenizers for the models listed in https://github.com/ludwig-ai/ludwig/blob/master/ludwig/schema/llms/base_model.py#L20 have the correct initialization.