Open rickgit16 opened 2 weeks ago
Hi, to turn on alibi in MPT, you'll want to not use the algorithm approach, but just specify it directly in the model architecture. Here is an example: https://github.com/mosaicml/llm-foundry/blob/c23be4ab9e146ff1064758a83fbe57c7d7a8e2ba/TUTORIAL.md#what-kinds-of-positional-embeddings-does-llm-foundry-support
Hi @dakinggg, thank you for reference. Do we still need to follow PR#820 for the setup?
Which part of that PR are you referring to? Just installing pip install .[gpu]
and specifying attn_impl: flash
should work fine
I am trying to pretrain a MPT model using llm-foundry using AliBi with flash attention. During pre training, I see the below warning -
I have followed PR#820 for alibi with FA2 for setup, and have used the following in pretrain yaml file -
Just to confirm alibi hasn't been used, I had converted the composer checkpoint to a HF one using
scripts/inference/convert_composer_to_hf.py
. I find theattn_config.alibi
flag is set toFalse
in theconfig.json
file.Some insights and direction on how to use alibi with flash attention 2 would be immensely helpful.