Closed sigridjineth closed 5 months ago
Hello @sigridjineth, in the FuseLLM project, please try using flash_attn version "flash_attn==0.2.8". Or you can change the code to use flash_attn_2.
trust_remote_code = False
tknz_trust_remote_code = False
use_fast = False
# Set RoPE scaling factor
config = transformers.AutoConfig.from_pretrained(
model_args.model_name_or_path,
cache_dir=training_args.cache_dir,
trust_remote_code=trust_remote_code
)
orig_ctx_len = getattr(config, "max_position_embeddings", None)
if orig_ctx_len and training_args.model_max_length > orig_ctx_len:
scaling_factor = float(math.ceil(training_args.model_max_length / orig_ctx_len))
config.rope_scaling = {"type": "linear", "factor": scaling_factor}
config.use_cache = False
compute_dtype = (
torch.bfloat16
if training_args.bf16
else (torch.float16 if training_args.fp16 else torch.float32)
)
# Load model and tokenizer
model = transformers.AutoModelForCausalLM.from_pretrained(
model_args.model_name_or_path,
config=config,
cache_dir=training_args.cache_dir,
use_flash_attention_2=True if training_args.flash_attn_transformers else False,
torch_dtype=compute_dtype,
trust_remote_code=trust_remote_code
)
tokenizer = transformers.AutoTokenizer.from_pretrained(
model_args.model_name_or_path,
cache_dir=training_args.cache_dir,
model_max_length=training_args.model_max_length,
padding_side="right",
trust_remote_code=tknz_trust_remote_code,
use_fast=use_fast,
)
tokenizer.pad_token = tokenizer.unk_token
I found that Get Error:use_cache is not supported happens when training, so that turning off Flash Attention works for me. How can I workaround this? Have you encountered this issue?
You can follow the code and set config.use_cache = False
to solve this error.
I am consistently getting
flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so
error when running deepspeed.