search

wejoncy / QLLM

A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ, and export to onnx/onnx-runtime easily.
Apache License 2.0
150 stars 15 forks source link

add macro GENERAL_TORCH to get rid of OptionalCUDAGuard #129

Closed wejoncy closed 5 months ago

wejoncy commented 5 months ago

OptionalCUDAGuard can't be used on different torch versions