Bert模型不支持scaled_dot_product_attention

[I-Good-Vegetable]

您好，我尝试运行run.sh结果出现bert模型不支持scaled_dot_product_attention的错误。这是什么导致的呢？谢谢~ 下面是run.sh的内容

export CUDA_VISIBLE_DEVICES=0,1
python3 run.py \
    --task_name UniRel \
    --max_seq_length 100 \
    --per_device_train_batch_size 12 \
    --per_device_eval_batch_size 12 \
    --learning_rate 3e-5 \
    --num_train_epochs 100 \
    --logging_dir ./tb_logs \
    --logging_steps 50 \
    --eval_steps 5000000 \
    --save_steps 5000 \
    --evaluation_strategy steps \
    --warmup_ratio 0.1 \
    --model_dir bert-base-cased \
    --output_dir ./output/nyt \
    --overwrite_output_dir \
    --dataset_dir data4bert \
    --dataloader_pin_memory \
    --dataloader_num_workers 4 \
    --lr_scheduler_type cosine \
    --seed 2023 \
    --do_test_all_checkpoints\
    --dataset_name nyt \
    --test_data_type unirel_span \
    --threshold 0.5 \
    --do_train

出现的error：

ValueError: BertModel does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention yet. Please request the support for this architecture: https://github.com/huggingface/transformers/issues/28005. If you believe this error is a bug, please open an issue in Transformers GitHub repository and load your model with the argument `attn_implementation="eager"` meanwhile. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="eager")`

[I-Good-Vegetable]

唔唔~得在model_transformer.py的第27行后面加上,attn_implementation='eager'变成

self.bert = BertModel.from_pretrained(model_dir, config=config, attn_implementation='eager')

要不咱给它加上呢

[wtangdev]

Hi，我当时用的是transformers == 4.12.5库，对里面的BertModel做了一些修改让他能够输出attention score，建议使用对应版本的transformers库，看看是否问题依旧？

[I-Good-Vegetable]

嗯嗯好的好的～谢谢您