Refactor gptq marlin - Githubissues

neuralmagic / nm-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

https://nm-vllm.readthedocs.io

Other

251 stars 10 forks source link

Refactor gptq marlin #365

Closed robertgshaw2-neuralmagic closed 4 months ago

robertgshaw2-neuralmagic commented 4 months ago

SUMMARY:

updates compressed-tensors and gptq_marlin to use process_weights_after_loading for repacking rather
factors out marlin utilities to be shared across gptq_marlin and compressed-tensors
adds integration test for W4A16 compressed tensors
removes duplicated code for test and actual code pathways