Added model_max_length as a tok parameter

mozilla-ai / lm-buddy

Your buddy in the (L)LM space.

Apache License 2.0

64 stars 3 forks source link

Added model_max_length as a tok parameter #113

Closed aittalam closed 2 months ago

aittalam commented 2 months ago

Since 4.40.0, transformers does not get model_max_tokens from the model's family dict but explicitly looks for this parameter in tokenizer_config.json. Not all models provide this, so one possibility is to explicitly allow users to provide this parameter in the config. This PR implements exactly this.

Tested with facebook/bart-large-cnn + pytest unit/integration tests.

aittalam commented 2 months ago

approving given the internal constraints; double check a few things before landing though. I'd suggest adding an extra test to the unit tests to check for it.

Thanks! I have two more PRs in queue:

bump of all libs (this will allow us, anmong other things, to run more recent models)
adding text generation pipeline + prompts

Will make unit tests for both and add one for this change too (explicitly passing a longer text as input and verifying it does not break with the new setup)