search

samiraabnar / attention_flow

215 stars 28 forks source link

Contradiction between public results and bert_example.ipynb #10

Open Bachstelze opened 6 months ago

Bachstelze commented 6 months ago

The left bar in figure 4 a) in the paper shows the relative predicted probability for the two possible pronouns, “his” and “her” with a correct prediction to "his" for the input "the author talked to sara about [MASK] book". bert_example.ipynb is giving the opposite result for that sentence: grafik

Bachstelze commented 6 months ago

This false result is also produced by the pipelined huggingface inference API: grafik

Bachstelze commented 6 months ago

With bert-base-case the results are also wrong: grafik Also without punctuation: grafik

HelloWorldLTY commented 4 months ago

I think it make senes because during the plot the authors only keep his and her to display. In the plot you showed, the probability of her is always higher than his.

Bachstelze commented 1 month ago

@HelloWorldLTY What do you mean by "during the plot the authors only keep his and her to display"? The contradicting result is only about this input sentence "the author talked to sara about [MASK] book".

Bachstelze commented 3 weeks ago

The reason for the different results is probably different model sizes. BERT-large gives the same result as in the paper: grafik