When trying to show attention score for each generated token using showAttention function, I find that each row of the decoder_attentions does not sum up to 1, which seems strange for the attention mechanism. Most values are very small. Not sure if it is a bug?
When trying to show attention score for each generated token using
showAttention
function, I find that each row of thedecoder_attentions
does not sum up to 1, which seems strange for the attention mechanism. Most values are very small. Not sure if it is a bug?