Monkey Patch Xformers use `past_key_value` but `use_cache` can't be `True`?

[fahadh4ilyas]

I'm confused with this script longchat/train/monkey_patch/llama_xformer_monkey_patch.py. I thought by not allowing use_cache parameter to True means past_key_value will not be used and not return. But, somehow past_key_value is still used but attn_weights is always None. Did you mean to set

assert not output_attentions, "xformer does not support output_attentions=True"

instead of

assert not use_cache, "xformer does not support use_cache=True"

[DachengLi1]

@fahadh4ilyas Sorry for the confusion. The codes are intended to be used with use_cache=False, output_attentions=False. We will clean the use_cache logic in the next refactoring. Thanks for letting us know!

[fahadh4ilyas]

So, past_key_value is not supported in xformers?

[DachengLi1]

Right, it is intended for Training purpose, where past_key_value is always not provided.

[fahadh4ilyas]

Okay, that makes sense. Which one is faster? Using flash attention or xformers?

[DachengLi1]

In my benchmark during experiments, they are the same. Xformer supports V100, but flash does not.

[fahadh4ilyas]

What about for inference? Doesn't inference need past_key_value? Does this mean xformers can't be used for inference?

[DachengLi1]

Yes, xformer and flash attention are not intended for inference in their most naive usage. We have more advanced system supporting this, and some models already been supported by this system in FastChat. LongChat is the next todo.

[fahadh4ilyas]

Okay right... Thank you for the answer...

[lucasjinreal]

@DachengLi1 Hi, so if one need support v100 training, it's better using xformers?

BTW, does it work if full params finetune with xformers or flashattn?