search

ggerganov / llama.cpp

LLM inference in C/C++
MIT License
60.95k stars 8.7k forks source link

Add Unigram tokenizer needed by T5 and FLAN-T5 model families #8089

Closed fairydreaming closed 3 days ago

fairydreaming commented 5 days ago

This is a second PR from a series of PRs adding support for T5 and FLAN-T5 models.

This PR adds implementation of the Unigram tokenizer used in T5 and FLAN-T5 models. It also adds T5 model architecture, tensors and model header parameters to allow testing the tokenizer with llama-tokenize command.