Can we change mistral attention bias=False to bias=config.attention_bias ？

[Minami-su]

just like llama attention

[amyeroberts]

@Minami-su Thanks for this feature request! Could you provide some more context on why you'd like to have this feature added?

In principle, it can be done by adding the config.attention_bias param which defaults to False. Care will need to be taken to make sure this is properly propagated and backwards compatible with any models which copy mistral's attention.

You'll also need to make sure that if config.attention_bias=True then the flash attention and SDPA attentions remain equivalent to the eager implementation.

[Minami-su]

The default setting for mistral is bias=False,so setting config.attention_bias=True is useless @amyeroberts

[amyeroberts]

@Minami-su If it's useless then there's no point in adding this config parameter

[Minami-su]

@Minami-su If it's useless then there's no point in adding this config parameter

I have a model https://huggingface.co/Minami-su/Qwen1.5-7B-Chat_mistral Because qwen attention has bias weight, but the model architecture is mistral,so....

[amyeroberts]

so setting config.attention_bias=True is useless

If this is useless then there's 0 value in being able to set this. If it's not useless, and you do want to add a config parameter to control the addition of bias, such as you've done in your custom model, then it's necessary to make sure the different attention layers remain equivalent.