search

Qualcomm-AI-research / gptvq

BSD 3-Clause Clear License
16 stars 1 forks source link

Exact hyperparameters #1

Open VityaVitalich opened 3 months ago

VityaVitalich commented 3 months ago

Dear maintainers,

I have couple questions according to hyperparameters.

1) What are the exact hyperparameters to run llama.py that are reported in Table2 for GPTVQ 2D for Llama2-7b? I am struggling to find them in paper and match with code.

2) there is a piece of code in VQQuantizer. It seems like if I am passing 2 wbit and 2 vqdim, then number of my centroids will be 16. If so, recalculating BPV, the first term is log_2(k) will be 4, that does not match with the paper 2.125 BPV. it seems like the only way to obtain 2.125 or 2.25 BPV with that piece of code is to make self.wbits = 2, therefore in case of vq_dim=2, we need to set wbits to 1?

def configure(self, wbits, **_):
        self.wbits = int(wbits * self.vq_dim)
        self.n_centroids = int(2**self.wbits)
Blank-z0 commented 3 weeks ago

Same question too -( I found the paper's explanation of many hyperparameters such as group size, block size, etc. to be very confusing. What is meaning of "group size" in gptvq? For example, for a llama-2 model, some matrices are 4096*4096, if columns-per-group=256 and groupsize=256, how to calculate "the number of columns in a group" (in alg1, line 2 in the paper)?