Enable the use_flash_attention configuration flag for Falcon models. When use_flash_attention is set to true the FalconAttention.forwad() method is replaced with a variant that uses Tri Dao's flash_attention instead of pytorch's scaled_dot_product_attention function.
At the moment the patch works only for falcon-7b but technically it will also work for falcon-40b with the right configuration. The falcon model situation is currently a bit messy: The Falcon model was recently added to Huggingface transformers (see PR transformers#24523) but the falcon models on the hugginface hub use still the code which is shipped together with the weights (a PR to change this was reverted). Falcon-7b and 40b use both slightly different code (which was unified in the HF transformers impl and can there be controlled via a configuration member called new_decoder_architecture see configuration_falcon.py#L65-L67). The HF Falcon impl uses different names in the configuration class, e.g. compare new configuration_falcon.py and old configuration_RW.py
HF Falcon implementation compatible model configurations can be found here:
7B: config.json
40B: config.json
Enable the
use_flash_attentionconfiguration flag for Falcon models. Whenuse_flash_attentionis set totruethe FalconAttention.forwad() method is replaced with a variant that uses Tri Dao's flash_attention instead of pytorch'sscaled_dot_product_attentionfunction.At the moment the patch works only for falcon-7b but technically it will also work for falcon-40b with the right configuration. The falcon model situation is currently a bit messy: The Falcon model was recently added to Huggingface transformers (see PR transformers#24523) but the falcon models on the hugginface hub use still the code which is shipped together with the weights (a PR to change this was reverted). Falcon-7b and 40b use both slightly different code (which was unified in the HF transformers impl and can there be controlled via a configuration member called
new_decoder_architecturesee configuration_falcon.py#L65-L67). The HF Falcon impl uses different names in the configuration class, e.g. compare new configuration_falcon.py and old configuration_RW.pyHF Falcon implementation compatible model configurations can be found here: 7B: config.json 40B: config.json