Pelochus / ezrknn-llm

Easier usage of LLMs in Rockchip's NPU on SBCs like Orange Pi 5 and Radxa Rock 5 series
Other
36 stars 3 forks source link

Change LLM templates with files #2

Open Pelochus opened 2 months ago

Pelochus commented 2 months ago

Essentially changing the LLM template dynamically at program execution. Add a parameter that reads a file and changes the corresponding variables. Especially useful if using various different LLMs. This needs some research first, as I know pretty much nothing about the templates.

Pelochus commented 2 months ago

Initial point of reference:

https://gist.github.com/av1d/ce58ef738902ec0365fe828720be31e5

Pelochus commented 2 months ago

Important link to this issue:

https://www.reddit.com/r/RockchipNPU/comments/1cpngku/rknnllm_v101_lets_talk_about_converting_and/

I could also add an option to not use files, use parameters or something. Perhaps reconverting everything with a modified tokenizer.json could be also a good option, for having the same template for every converted model

80Builder80 commented 2 months ago

We anticipated better with an update. Instead we get useless.