-
I would like to know what should it be the minimum requirement specifications for running 7B models. I have this configuration and only can run 1.5B with average throughput 8.1 token/s.
![image](http…
-
`llm_inference` on local is throwing the below error for `gemma-2b-it-cpu-int8.bin` only `gpu` backend type is supported?
```
calculator_graph.cc:892] INVALID_ARGUMENT: CalculatorGraph::Run() fail…
-
Hi
How can I assign multiple cpus to the training process?
Thanks a lot.
-
1) microsoft/Phi-3-small-128k-instruct is not running on cpu (ICELAKE or Graviton)
the script which i was using :
![Screenshot 2024-07-26 165001](https://github.com/user-attachments/assets/e0b2…
-
**Describe the bug**
Awq model export for cpu is not supported
**To Reproduce**
python3 builder.py -i awq_model_dir -o output_folder -p int4 -e cpu -c cache_dir
**Screenshots**
![屏幕截图 2024-08…
-
Does oneflow support model offloading like pipe.to('cpu') while all the graphs are being loaded ?
-
### What happened?
On compiling a model with int8 quantization one of the dispatches fails to compile with the following error:
```
error: One or more operations with large vector sizes (16384 …
-
After updated tgi version to
ghcr.io/huggingface/text-generation-inference:latest-intel-cpu
The codegen test failed with the following 2 MODELs:
ise-uiuc/Magicoder-S-DS-6.7B
m-a-p/OpenCodeInterpr…
-
I have three machines:
- "New" is a new heavy compute server; it is the fastest machine, with oodles of memory and cores and SSD in a fast setup
- "Old" is an aging compute server; it is very fast…
-
### Feature request
The problem I'm facing, is that a running model within an extension, even on the background script (service worker) - it's blocking other actions within the extension. This mean…