Closed dwadden closed 4 months ago
Per @hamishivi comment on PR #121, add an option to specify a separate tokenizer_revision if requested. This seems like an unusual use case, so the code throws a warning if the model and tokenizer have different revisions.
tokenizer_revision
Per @hamishivi comment on PR #121, add an option to specify a separate
tokenizer_revisionif requested. This seems like an unusual use case, so the code throws a warning if the model and tokenizer have different revisions.