Fix linear bias of qkv layers in models.

PygmalionAI / aphrodite-engine

PygmalionAI's large-scale inference engine

https://pygmalion.chat

GNU Affero General Public License v3.0

606 stars 78 forks source link

Fix linear bias of qkv layers in models. #430

Closed sgsdxzy closed 3 weeks ago

sgsdxzy commented 3 weeks ago

Previous the weight shapes of ColumnParallelLinear are incorrect for q,k,v layers of some models, because the output size is divided by tp size twice.