Epic: Auto add tokenizer.model

[hahuyhoang411]

Problem

Sometimes the model HF repo doesn't have tokenizer.model itself which cause the bug missing tokenizer.

Recommendation

@nguyenhoangthuan99 : Add 1 more step in CI converter

https://github.com/ggerganov/llama.cpp/issues/2443

Tasks

• [x] Testing manual convert
• [x] Add to CI

[hahuyhoang411]

I think the problem is related to our current llama.cpp version

https://github.com/ggerganov/llama.cpp/issues/6419

[hahuyhoang411]

I convert the model using our internal cluster with latest llama.cpp version and it works fine

Can you help me to bump the version cc @nguyenhoangthuan99

[hahuyhoang411]

Problem solved with version bump

[Srujan-D]

Hi @hahuyhoang411

I am trying to add Florence model support to llama.cpp, and am facing the tokenizer.model issue. Can you please help out with what steps I need to follow to get tokenizer.model from tokenizer.json?

Thanks!