How do I output the attention from the model?

MAGICS-LAB / DNABERT_2

[ICLR 2024] DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genome

Apache License 2.0

254 stars 59 forks source link

Open jkb-ag opened 6 months ago

jkb-ag commented 6 months ago

Hi, When running dnabert2, I'm trying to output not only the CLS and hidden states but also the attentions.

I tried:

`db2_model = AutoModel.from_pretrained(dirname, output_attentions=True, trust_remote_code=True)

output = db2_model(input_ids) `

but the output is only a tuple of length 2 (the hidden states and the CLS tensor).

Any help would be much appreciated!

Thanks!

ZoeyXiang1202 commented 5 months ago

Hi,

I also have the same question. Could you please show where the finetuned model weight is saved?

Thanks a lot!