inference/cli_demo_quantization.py does not work with torchao from current git tree anymore
Traceback (most recent call last):
File "/home/x/CogVideo/inference/cli_demo_quantization.py", line 26, in
from torchao.float8.inference import ActivationCasting, QuantConfig, quantize_to_float8
ImportError: cannot import name 'ActivationCasting' from 'torchao.float8.inference'
backface
commented
1 week ago
inference/cli_demo_quantization.py does not work with torchao from current git tree anymore
Traceback (most recent call last): File "/home/x/CogVideo/inference/cli_demo_quantization.py", line 26, in
from torchao.float8.inference import ActivationCasting, QuantConfig, quantize_to_float8
ImportError: cannot import name 'ActivationCasting' from 'torchao.float8.inference'
it seems it got removed here: https://github.com/pytorch/ao/commit/848e123e37df7e7033f26619b02562525404c2b5
Is there any up-to-date example of how to do inference with quantization?