feat(model): llama load tokenizer

boss-chanon commented 1 year ago

Why this PR

load llama tokenizer for use in lanta front end

Close #

[ ] PR should be in the Naming convention
[ ] Assign yourself in to Assigneees
[ ] Tag related issues
[ ] Constants name should be ALL_CAPITAL, function name should be snake_case, and class name should be CamelCase
[ ] complex function/algorithm should have Docstring
[ ] 1 PR should not have more than 200 lines changes (Exception for test files). If more than that please open multiple PRs
[ ] At least PR reviewer must come from the task's team (model, eval, data)

boat1603 commented 1 year ago

Why we need this PR?

boss-chanon commented 1 year ago

Why we need this PR?

we can't load tokenizer in backend of lanta we must load tokenizer and save it in frontend before use tokenizer