Open NirantK opened 6 months ago
See Static & Dynamic quantization here: https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization
From @Xenova, this script traverses the graph and collects operators for quantization https://github.com/xenova/transformers.js/blob/main/scripts/convert.py
Consider this model from Xenova, there is a quantized model which is 120M instead of the 440-450M which I get from O3 quantization from Optimum.
Compare if the quantized model is as good as the 450M, O3 with an atol of 1e-3 and an O2 of 1e-4 — or there is something else happening there?