ggerganov / llama.cpp

LLM inference in C/C++
MIT License
65.59k stars 9.41k forks source link

Added support for Viking pre-tokenizer #8135

Closed kustaaya closed 3 months ago

kustaaya commented 3 months ago

Implemented suggested changes from discussion in pull request #7328 by @akx . Added pre-tokenizer support for Viking-models.

Tokenizer is the same for Viking-7B and its 13B and 33B siblings. Fixes issue #7309

slaren commented 3 months ago

The changes look correct me, but there are conflicts that need to be resolved before this can be merged. I think it is just because llama.h and llama.cpp were moved to a different directory.

kustaaya commented 3 months ago

The changes look correct me, but there are conflicts that need to be resolved before this can be merged. I think it is just because llama.h and llama.cpp were moved to a different directory.

Thanks! I updated the pull request to fix the conflicts.