mudler
commented
4 months ago cc @fakezeta can you confirm with your setup? I didn't gave a shot at OpenVINO locally here yet as I'm using sycl, otherwise will have a look later in the weekend
Open gericho opened 4 months ago
mudler
commented
4 months ago cc @fakezeta can you confirm with your setup? I didn't gave a shot at OpenVINO locally here yet as I'm using sycl, otherwise will have a look later in the weekend
fakezeta
commented
4 months ago Hi all, I'm currently on vacation with my family this week and as such I've limited capabilities to support (I'm only with mobile phone).
From the last log I saw no error. @gericho can you kindly do a test adding "stream": true to the json?
In the code streaming and no streaming generation are handled differently.
localai-bot
commented
4 months ago Certainly, I can try that. However, to provide a more comprehensive solution, could you please provide the specific code or configuration file where the "stream" option should be added? I don't want to make unnecessary changes or modify the wrong file. Thank you.
gericho
commented
4 months ago So I tried with the stream option and got letter by letter inference as expected. The CPU usage is around 10-15%, the performance is way worse than @fakezeta showed here on reddit.
Here the command with the "stream" option I just used:
curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{ "stream": true, "model": "starling-openvino", "prompt": "A long time ago in a galaxy far, far away", "temperature": 0.7 }
See here fakezeta CPU specs See here my CPU specs
@fakezeta said his localai is running on a VM, mine is running on Proxmox passing the iGPU to the LXC then to the Docker Container. So theoretically, seen the CPUs specs, it have to run almost at the same speed if not faster on the i3 N300.
Here my LXC status when working, the RAM usage way lower than the model size itself :
root@localai:~# vainfo
error: can't connect to X server!
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.1 ()
vainfo: Supported profile and entrypoints
VAProfileNone : VAEntrypointVideoProc
VAProfileNone : VAEntrypointStats
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSliceLP
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSliceLP
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSliceLP
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointEncSliceLP
VAProfileVP9Profile1 : VAEntrypointVLD
VAProfileVP9Profile1 : VAEntrypointEncSliceLP
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointEncSliceLP
VAProfileVP9Profile3 : VAEntrypointVLD
VAProfileVP9Profile3 : VAEntrypointEncSliceLP
VAProfileHEVCMain12 : VAEntrypointVLD
VAProfileHEVCMain422_10 : VAEntrypointVLD
VAProfileHEVCMain422_12 : VAEntrypointVLD
VAProfileHEVCMain444 : VAEntrypointVLD
VAProfileHEVCMain444 : VAEntrypointEncSliceLP
VAProfileHEVCMain444_10 : VAEntrypointVLD
VAProfileHEVCMain444_10 : VAEntrypointEncSliceLP
VAProfileHEVCMain444_12 : VAEntrypointVLD
VAProfileHEVCSccMain : VAEntrypointVLD
VAProfileHEVCSccMain : VAEntrypointEncSliceLP
VAProfileHEVCSccMain10 : VAEntrypointVLD
VAProfileHEVCSccMain10 : VAEntrypointEncSliceLP
VAProfileHEVCSccMain444 : VAEntrypointVLD
VAProfileHEVCSccMain444 : VAEntrypointEncSliceLP
VAProfileAV1Profile0 : VAEntrypointVLD
VAProfileHEVCSccMain444_10 : VAEntrypointVLD
VAProfileHEVCSccMain444_10 : VAEntrypointEncSliceLPPlease note, it is running over 6000 tokens and still counting... how can I limit that?
EDIT: Just finished, here the log
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"J"}],"usage":{"prompt_tokens":0,"completion_tokens":7061,"total_tokens":7061}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"o"}],"usage":{"prompt_tokens":0,"completion_tokens":7062,"total_tokens":7062}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"h"}],"usage":{"prompt_tokens":0,"completion_tokens":7063,"total_tokens":7063}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"n"}],"usage":{"prompt_tokens":0,"completion_tokens":7064,"total_tokens":7064}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":" "}],"usage":{"prompt_tokens":0,"completion_tokens":7065,"total_tokens":7065}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"W"}],"usage":{"prompt_tokens":0,"completion_tokens":7066,"total_tokens":7066}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"i"}],"usage":{"prompt_tokens":0,"completion_tokens":7067,"total_tokens":7067}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"l"}],"usage":{"prompt_tokens":0,"completion_tokens":7068,"total_tokens":7068}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"l"}],"usage":{"prompt_tokens":0,"completion_tokens":7069,"total_tokens":7069}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"i"}],"usage":{"prompt_tokens":0,"completion_tokens":7070,"total_tokens":7070}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"a"}],"usage":{"prompt_tokens":0,"completion_tokens":7071,"total_tokens":7071}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"m"}],"usage":{"prompt_tokens":0,"completion_tokens":7072,"total_tokens":7072}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":"s"}],"usage":{"prompt_tokens":0,"completion_tokens":7073,"total_tokens":7073}}
local-ai-1 |
local-ai-1 | 6:09PM DBG completion streaming sending chunk: {"choices":[{"index":0,"finish_reason":"","text":":"}],"usage":{"prompt_tokens":0,"completion_tokens":7074,"total_tokens":7074}}UPDATE: As per @fakezeta advice, since the performance on the smaller i3 N300 with an int8 model gives not usable results, I just tried, and got way faster results, with the Starling int4 model! I'm still not able to limit the max token output BTW
Here the link to the model used fakezeta/Starling-LM-7B-beta-openvino-int4
@fakezeta advice also to try a OLLAMA or llama.cpp GGUF model, that could be faster. I will update this post eventually.
fakezeta
commented
4 months ago @mudler there could be something in the non-streaming generation since streaming is working.
I'll look at it when I get home in the weekend.
@gericho Ollama is using the same llama.cpp the LocalAI is using for GGUF but on your machine I expect OpenVINO 4bit quantization to be faster.
i3-N300 has a 32 EU GPU like mine but could be power limited since it has a TDP of only 7 watt vs 65 of mine.
Ram speed is also important, what kind of Ram do you have? I use DDR5-5600
gericho
commented
4 months ago @mudler there could be something in the non-streaming generation since streaming is working.
I'll look at it when I get home in the weekend.
@gericho Ollama is using the same llama.cpp the LocalAI is using for GGUF but on your machine I expect OpenVINO 4bit quantization to be faster.
i3-N300 has a 32 EU GPU like mine but could be power limited since it has a TDP of only 7 watt vs 65 of mine.
Ram speed is also important, what kind of Ram do you have? I use DDR5-5600
32GB DDR5 4800MHz CL40
@fakezeta thank you for pointing out some more OpenVINO GGUF models from https://huggingface.co/helenai
EDIT: just to be clear, OpenVINO does not support GGUF. Is llama.cpp that uses GGUF models and has sycl acceleration for iGPU. So if I'm correct, only LocalAI docker images like this one quay.io/go-skynet/local-ai:master-sycl-f16-ffmpeg have acceleration support to GGUF models in OpenVINO.
gericho
commented
4 months ago Just an update with some performance data on the i3 N300 8 cores 32 GB single-channel 4800Mhz RAM (7W TDP), it may be fair to check also if possible to tweak the bios since this machine was born fanless but can actually accept a 70mm fan:
Docker Image used: quay.io/go-skynet/local-ai:master-sycl-f16-ffmpeg
Model fakezeta/Starling-LM-7B-beta-openvino-int4
trial 1 : 611 tokens 33 secs 18,5tk/s
trial 2 : 517/30 17.23tk/s
trial 3 : 584/32 18,25tk/s
around 18tk/sModel fakezeta/Starling-LM-7B-beta-openvino-int8
trial 1 : 583 tokens 40 secs 14,5tk/s
trial 2 : 539/38 14,18tk/s
trial 3 : 618/41 15,07tk/s
around 14,5tk/sNote these trials were timed manually, so the whole chain is evaluated: Prompt evaluation, Token generation and Token decoding. Note also, this machine is a Proxmox 8 host, and it runs simultaneously a Home Assistant VM and some LXC, 3 LCXs are sharing OpenVINO and VAAPI backends too!
The idea is to give Home Assistant a conversation-agent, so it's important the model can handle functions rendering but not necessarily (or mainly/primarly) chitchat responses. So the given models mentioned above are way overkill for this IMHO.
fakezeta
commented
4 months ago Back at home and working on this thanks to @gericho availability on direct chat. Happy to see that performance are consistent with my testing HW and it confirms my idea of OpenVINO as a viable solution for edge use cases.
I tryed the curl command above:
curl http://localhost:8080/v1/completions -H "Authorization: Bearer REDACTED" -H "Content-Type: application/json" -d '{ "stream": false, "model": "gpt-3.5-turbo", "max_tokens": 15, "prompt": "Why the grass is green?", "temperature": 0.7 }'
output is
{"created":1713541361,"object":"text_completion","id":"bbea4f56-de56-495c-b696-60467772558d","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"stop","text":"\n\nThe grass appears green due to the presence of a pigment called"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
with this model definition file:
name: gpt-3.5-turbo
backend: transformers
parameters:
model: fakezeta/Starling-LM-7B-beta-openvino-int8
context_size: 8192
threads: 6
f16: true
type: OVModelForCausalLM
stopwords:
- <|end_of_turn|>
- <|endoftext|>
prompt_cache_path: "cache"
prompt_cache_all: true
template:
chat_message: |
{{if eq .RoleName "system"}}{{.Content}}<|end_of_turn|>{{end}}{{if eq .RoleName "assistant"}}<|end_of_turn|>GPT4 Correct Assistant: {{.Content}}<|end_of_turn|>{{end}}{{if eq .RoleName "user"}}GPT4 Correct User: {{.Content}}{{end}}
chat: |
{{.Input}}<|end_of_turn|>GPT4 Correct Assistant:
completion: |
{{.Input}}The Authorization: Bearer header can be removed, I use it because I have LocalAI exposed to internet.
So both non-streaming and max tokens seems to work.
I asked @gericho to try this command and report feedback.
nickp27
commented
4 months ago Just a comment to say that progress on Openvino has brought me back to LocalAI, so thank you for that. I had experimented with other intel openAI api solutions like BigDL-LLM with FastChat and IPEX/NeuralChat but usually abandoned them for performance or support reasons.
Got it working today with Starling-LM-7B-beta-openvino-int4, and solid performance on an i5-1135G7.
Currently testing with aless2212/Mistral-7B-v0.2-openvino-int4-cpu - performance seems as good, but cannot for the life of me make the chat template work - keep getting non-sense responses. Will keep trying.
fakezeta
commented
4 months ago Thank you for your kind words, which truly warm my heart, especially considering the late-night hours I’ve spent on the transformer backend.
Regarding the chat template issues, the pull request #2090 should provide a solution. You will now be able to utilize the chat template included in the tokenizer_config.json file.
The model definition has been streamlined to the following:
name: mistral
backend: transformers
parameters:
model: aless2212/Mistral-7B-v0.2-openvino-int4-cpu
context_size: 8192
threads: 6
f16: true
type: OVModelForCausalLM
template:
use_tokenizer_template: trueJust a comment to say that progress on Openvino has brought me back to LocalAI, so thank you for that. I had experimented with other intel openAI api solutions like BigDL-LLM with FastChat and IPEX/NeuralChat but usually abandoned them for performance or support reasons.
Got it working today with Starling-LM-7B-beta-openvino-int4, and solid performance on an i5-1135G7.
Currently testing with aless2212/Mistral-7B-v0.2-openvino-int4-cpu - performance seems as good, but cannot for the life of me make the chat template work - keep getting non-sense responses. Will keep trying.
nickp27
commented
4 months ago Just updated to the latest docker image. Suspect it might be an issue with the model itself.
Config:
name: mistral-openvino
backend: transformers
parameters:
model: aless2212/Mistral-7B-v0.2-openvino-int4-cpu
context_size: 8192
threads: 8
f16: true
type: OVModelForCausalLM
prompt_cache_path: "cache"
prompt_cache_all: true
template:
use_tokenizer_template: trueCurl test:
curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{ "stream": false, "model": "mistral-openvino", "max_tokens": 20, "prompt": "Tell me a story", "temperature": 0.7 }'
Response:
{"created":1713694939,"object":"text_completion","id":"a53748bc-c52a-48c8-8f50-a3115fd8fd01","model":"mistral-openvino","choices":[{"index":0,"finish_reason":"stop","text":"The as in redet #A.W. The more… Universityof1 # is a."}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}@nickp27 the PR is in his journey to be merged: I just opened it yesterday 😄
I tested this model for you with my local build and I think there is something not working with it:
curl http://localhost:8080/v1/completions -H "Authorization: Bearer c969ac29" -H "Content-Type: application/json" -d '{ "stream": false, "model": "mistral-openvino", "max_tokens": 100, "prompt": "Tell me a story", "temperature": 0.7 }'
{"created":1713696507,"object":"text_completion","id":"b7929f05-2012-4300-b2ba-6aee6e7df81e","model":"mistral-openvino","choices":[{"index":0,"finish_reason":"stop","text":"in #101 ( Question 391 ( User Question : L3, #17390020. The Goddubiquest_120,)\n, 2010riding for you\n Q: Q: #1 Question: #w by by bytown_ Question marks of Question of course Question Marker ( Questionnai T1 # Q3 # QEar- Question. Question"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
Also using the mistral template explicitly
template:
chat: &chat |
[INST] {{.Input}} [/INST]
completion: *chatgives garbage:
{"created":1713697087,"object":"text_completion","id":"34a6c693-635b-4c29-a22b-c3c0109485c4","model":"mistral-openvino","choices":[{"index":0,"finish_reason":"stop","text":"\nOnce orheresaid: weit, eranotherwiseashe mostimmediate. rodznaive, sostra in a new!\n QEP: a20- Question: mice,* Question: I's Question: the numbers Question—\n QEs more siendo_ Question: Q miemage^{ Question Question of Question: Q8 #dan a Question: Q Fuß est- Q: Q\\ Question: Q6 \" Question: Question "}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
nickp27
commented
4 months ago Good shout - i'll go back to your Starling model for now.
@nickp27 the PR is in his journey to be merged: I just opened it yesterday 😄
I tested this model for you with my local build and I think there is something not working with it:
curl http://localhost:8080/v1/completions -H "Authorization: Bearer c969ac29" -H "Content-Type: application/json" -d '{ "stream": false, "model": "mistral-openvino", "max_tokens": 100, "prompt": "Tell me a story", "temperature": 0.7 }'
{"created":1713696507,"object":"text_completion","id":"b7929f05-2012-4300-b2ba-6aee6e7df81e","model":"mistral-openvino","choices":[{"index":0,"finish_reason":"stop","text":"in #101 ( Question 391 ( User Question : L3, #17390020. The Goddubiquest_120,)\n, 2010riding for you\n Q: Q: #1 Question: #w by by bytown_ Question marks of Question of course Question Marker ( Questionnai T1 # Q3 # QEar- Question. Question"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}Also using the mistral template explicitly
template: chat: &chat | [INST] {{.Input}} [/INST] completion: *chatgives garbage:
{"created":1713697087,"object":"text_completion","id":"34a6c693-635b-4c29-a22b-c3c0109485c4","model":"mistral-openvino","choices":[{"index":0,"finish_reason":"stop","text":"\nOnce orheresaid: weit, eranotherwiseashe mostimmediate. rodznaive, sostra in a new!\n QEP: a20- Question: mice,* Question: I's Question: the numbers Question—\n QEs more siendo_ Question: Q miemage^{ Question Question of Question: Q8 #dan a Question: Q Fuß est- Q: Q\\ Question: Q6 \" Question: Question "}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
fakezeta
commented
4 months ago PR merged, it's in the latest docker build local-ai:master-sycl-f16-ffmpeg
On my HF you can find some other models like WizardLM2 and also Llama3. Beware that I couldn't get Llama3 to work properly: still have to understand if it's a configuration issue (like stopwords and chat template) or a transformer library issue fixed in 4.40 that I cannot use since optimum-intel supports up to 4.39. If you have time to investigate you're welcome 😄
Since this issue it's becoming more a general discussion on OpenVINO why don't close this issue and continue on a discussion thread?
HI! I'm trying to use the OpenVINO support on my machine, I was able to make the model download automatically but I got the error showed down in the log attached.
LocalAI version: quay.io/go-skynet/local-ai:master-sycl-f16-ffmpeg
Environment, CPU architecture, OS, and Version: i3-N300 32GB RAM (it is a N300 not a N305 as showed in the debug logs)
Describe the bug
To Reproduce
Docker Compose used
Modle YAML defined in /root/models
Expected behavior Model loads successfully and is possible to generate
Logs
Additional context Query Used