[Feature Request]: Using llama.cpp as a translator

[allegro0132]

What would your feature do?

Hazukiaoi mentions Baichun2 is a state-of-the-art translator between Japanese and Chinese. https://github.com/zyddnys/manga-image-translator/issues/460

I think we could implement llama.cpp as the backend of the translator, which provides the bridge to a series of Large Language models. Furthermore, SakuraUmi (https://github.com/pipixia244) released a fine-tuned Baichun2 model named Sakura13B. Sakura13B is designed for translation workloads like Galgame and Light Novel, I confirm it's also suitable for comic books.

Here I have implemented an LLM-based OfflineTranslator named SakuraTranslator. It uses llama.cpp as the backend and is built with Metal support. Commit link: https://github.com/allegro0132/manga-image-translator/commit/167e335318b2a4162a2d5fc0b9469699d04dba0b

[ywtse]

some feedback on using this model

model been renamed in huggingface, to edit

manga_translator/translators/sakura.py

--- "url": "https://huggingface.co/SakuraLLM/Sakura-13B-LNovel-v0.8-GGUF/resolve/main/sakura-13b-lnovel-norm-Q5_K_M.gguf" +++ "url": "https://huggingface.co/SakuraLLM/Sakura-13B-LNovel-v0.8-GGUF/resolve/main/sakura-13b-lnovel-v0.8-Q5_K_M.gguf"

--- _SAKURA_MODEL_NAME = 'sakura-13b-lnovel-norm-Q5_K_M.gguf' +++ _SAKURA_MODEL_NAME = 'sakura-13b-lnovel-v0.8-Q5_K_M.gguf'

--- n_gpu_layers = -1 if device == "mps" else 0 +++ n_gpu_layers = -1

https://github.com/abetlen/llama-cpp-python#installation-with-specific-hardware-acceleration-blas-cuda-metal-etc you can enable hardware acceleraton other then "mps" by "BLAS" tested on my iGPU730 with CLBlast and RTX4060Ti with cuBLAS

CMAKE_ARGS="-DLLAMA_CLBLAST=on" pip install llama-cpp-python --upgrade --force-reinstall ## - for opencl CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python --upgrade --force-reinstall ## - for cuda

maybe someone can test an AMD with "hipBLAS" ??

testing environment: OS : Ubuntu 22.04 x86_64 CPU : Intel i5-12400 GPU : Intel UHD Graphics 730 & Nvidia RTX4060Ti VRAM : 16GB SysRAM : 96GB

[SpaceThing]

使用https://github.com/allegro0132/manga-image-translator/commit/167e335318b2a4162a2d5fc0b9469699d04dba0b 替换了源文件，并且修改了部分代码，希望能使用提前下载好的"sakura-13b-lnovel-v0.9b-Q5_K_M.gguf"模型（huggingface下载太慢了），但是在web模式的列表里找不到sakura

replaced original files with https://github.com/allegro0132/manga-image-translator/commit/167e335318b2a4162a2d5fc0b9469699d04dba0b, and modified some code to use my pre-downloaded model file "sakura-13b-lnovel-v0.9b-Q5_K_M.gguf", but Sakura did not show up in the translator list

修改后的sakura.py (modified sakura.py file) sakura.py.zip

[yujianke100]

单独fork了一份并完善了点楼上两位的内容https://github.com/yujianke100/manga-image-translator