Implement on CPU

[minhduc01168]

After fine tuning the GPT-OCR 2.0 model in Vietnamese language, I obtained a model deployed on the GPU. Can you tell me where to start with the steps to implement it on CPU? I thank you very much.

[1694439208]

按照README.md说明转换模型为gguf

[minhduc01168]

@1694439208 Can you tell me more clearly? I haven't determined what I need to do? I am very grateful to you

[joeqi0370]

Step 1. Convert to gguf Please install python 3.10 and llama.cpp first

conda create -n llama.cpp python=3.10 -y
conda activate llama.cpp

git clone https://github.com/ggerganov/llama.cpp.git
pip install -r llama.cpp/requirements.txt

https://github.com/ggerganov/llama.cpp/tree/master/gguf-py, install the latest gguf, to support minicpm3

pip uninstall gguf
cd llama.cpp/gguf-py
pip install --editable .

cd [to another folder]
git clone https://github.com/1694439208/GOT-OCR-Inference.git
pip install tiktoken

please download all model files in https://modelscope.cn/models/stepfun-ai/GOT-OCR2_0/files to a local folder module-GOT-OCR2_0_modelscope.cn

python GOT-OCR-Inference/convert_hf_to_gguf.py --model module-GOT-OCR2_0_modelscope.cn --outfile got_ocr2_16.gguf --outtype f16

Step 2: try to use model in CPU

Note: you need gcc gcc-c++5.0+ to install llama-cpp-python, please refer to https://stackoverflow.com/questions/36327805/how-to-install-gcc-5-3-with-yum-on-centos-7-2

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
pip install llama-cpp-python

after installing all the packages, enter GOT-OCR-Inference and run : python llama2.py

Hope these steps helps you.

[minhduc01168]

@joeqi0370 Do you know how I can input the model with a png,jpg format image? I am so grateful and grateful to you