-
from the issue "https://developer.apple.com/forums/thread/740518 how do we use the computational power of A17 Pro Neural Engine?"
I learn that if i want to inference my mlmodel on my ipad pro with …
-
## 👉 [Please follow one of these issue templates](https://github.com/pytorch/fairseq/issues/new/choose) 👈
Note: to keep the backlog clean and actionable, issues may be immediately closed if they do…
-
### Feature request
In the quantization procedure for qlora, there is the 'nf'4 storage datatype and the compute datatype (in the paper bfloat16 which is the original)(please refer to the image). The…
-
https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/
>High-performance INT8 inference kernels are …
-
Hello,
Is it possible to obtain a quantized .tflite version of YOLO v3 / YOLO Tiny v3 to do INT8 inference with the tools in this repository? I've tried using TensorFlow Lite's official tool, `toco`,…
-
Below is the list of issues we are hitting when running 25+ models end to end using onnx mode (onnx export/import -> torch-mlir -> iree-compile -> iree-runtime). You can find the models that lead to e…
-
Hello everyone, we are seeing slower than expected inference times on one of our CPU node with Intel(R) Xeon(R) Platinum 8362 CPU @ 2.80GHz with following instruction sets:
```
fpu vme de pse tsc…
-
Would be great if someone could look into implementing this particular version of int8 for serving LLMs.
https://huggingface.co/blog/hf-bitsandbytes-integration
-
您好,请问您提到的“Comparisons of PSNR performance and the inference latency of different models. The inference latency is tested on Dimensity 8100 SoC, NNAPI driver, INT8 precision and upsampling from 360 × 6…
-
Hi, I read the docs about `zero_quant`, but it seems to require extra training.
And in `deepspeed.init_inference`, the `dtype` can be set to int8, but the code does nothing for int8. https://github…