search

Vahe1994 / SpQR

Apache License 2.0
515 stars 40 forks source link

How evaluation being done without storing quantized weights? #24

Closed tahmiddialpad closed 11 months ago

tahmiddialpad commented 1 year ago

I would like to know how the quantized model's performance is evaluated while keeping this line blank?

How are the quantized weights being considered while evaluating the performance of the model since the value of quantizers here is empty?

poedator commented 1 year ago

Hello, @tahmiddialpad,

Thank you for interest to SpQR compression.

When the quantize() function is called in line 217 it quantizes and then immediately dequantizes the model weights. See lines 178-189 in spqr_engine.py(). The line that you mentioned would return quantizers necessary to save the compressed model. This saving functionality is not yet implemented.