Minimum code to run an LLM model from HuggingFace with OpenVINO

Programs / Files

#	file name	description
1	download_model.py	Download a LLM model, and convert it into OpenVINO IR model
2	inference.py	Run an LLM model with OpenVINO. One of the most simple LLM inferencing code with OpenVINO and the `optimum-intel` library.
3	inference-stream.py	Run an LLM model with OpenVINO and `optimum-intel`. Display the answer in streaming mode (word by word).
4	inference-stream-openvino-only.py	Run an LLM model with only OpenVINO. This program doesn't require any DL frameworks such as TF or PyTorch. Also, this program doesn't even use the '`optimum-intel`' library or HuggingFace tokenizers to run. This program uses a simple and dumb tokenizer (that I wrote) instead of HF tokenizers. Try swapping the tokenizer to HF tokenizer in case you see only garbage text from the program (uncomment `AutoTokenizer` and comment out `SimpleTokenizer`)
5	inference-stream-openvino-only-greedy.py	Same as program #4 but uses 'greedy decoding' instead of sampling. This program generates fixed output text because it always picks the most probability token ID from the predictions (=greedy decoding).
6	inference-stream-openvino-only-stateless.py	Same as program #4 but supports STATELESS models (which does not use the internal state variables to keep KV-cache values inside of the model) instead of stateful models.

Note: Converting LLM model requires a large amount of memory (>=32GB).

python -m venv venv
venv\Scripts\activate
python -m pip install -U pip
pip install -U setuptools wheel
pip install -r requirements.txt

Run inference

python inference.py
# or
python inference-stream.py

stream.gif GitHub repository

Following web sites are also infomative and helpful for optimum-intel users.