search

JiaDingCN / GeminiFusion

MIT License
41 stars 5 forks source link

Is the calculation of the value vector incorrect? #10

Closed WhenMelancholy closed 3 weeks ago

WhenMelancholy commented 3 weeks ago

image Should this line of code be changed to noise_v = self.v_noise.weight[0] + v ?

JiaDingCN commented 3 weeks ago

The noise is added into the query itself to make the noised self attention.