Optional `bias` for qwen2 model - Githubissues

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

https://huggingface.co/transformers

Apache License 2.0

132.01k stars 26.3k forks source link

Optional `bias` for qwen2 model #32892

Open wavy-jung opened 3 weeks ago

wavy-jung commented 3 weeks ago

Feature request

bias of linear layers in qwen2 model is hard coded as following:

It would be good to make bias optionally configurable through a config file to ensure compatibility with the latest models. (e.g. llama)

Motivation

bias is optional in llama model as following:

https://github.com/huggingface/transformers/blob/85345bb439652d3f03bb4e123cef7a440f2ba95b/src/transformers/models/llama/modeling_llama.py#L286-L288

Your contribution

I'll submit PR for this feature

amyeroberts commented 3 weeks ago

cc @ArthurZucker

ArthurZucker commented 3 weeks ago

Answered on the PR~