Strange visual attention results

[mojesty]

Hello @DeepRNN ! I took a look at attentions that model generates in test mode. I did the following: in base_model.py:200 i changed the code as following

                memory, output, scores, attentions = sess.run(
                    [self.memory, self.output, self.probs, self.attentions],
                    feed_dict = {self.contexts: contexts,
                                 self.last_word: last_word,
                                 self.last_memory: last_memory,
                                 self.last_output: last_output})

So after that, every attentions array has the shape batch_size, 196, beam_size and for simplicity I set beam_size=1 when testing. Next, I simply stack all attentions in one numpy array and visualize its content. I found two concerns:

1. Attentions maps for different tokens vary negligibly (for 1.jpg the maximum difference between 1st and 2nd tokens is ~e^-9).
2. The maps itself looks rather strange. For the test image with bus:

However, the caption of the image is both grammatically and semantically correct. I would like to discuss these results.

[JiayunLi]

Yeah. I encountered the same problem. I find that the self.attentions is only defined in the train mode, so I also add self.attentions in the test mode. However, the difference between attention maps is still negligible.

[mojesty]

May this occur because of some mistake that makes attention shared across all timestamps? 1e-9 difference can be explained them. If so, how can it be fixed?

[RoronoaZA]

@JiayunLi I think self.attentions is only used to be shown.

[JiayunLi]

@mojesty I haven't figured it out.

[JiayunLi]

@RoronoaZA Yeah. Are you able to get reasonable attention maps for evaluation images?

[minizon]

Same problem

[bright1993ff66]

Hi @mojesty , how long does it take to train one epoch? What machines did you use? Thank you!

[chansongoal]

Has anyone solve the problem? Anyone can help me? Thank you all.

[Fanshia]

The attention layer defined in model.py is: image context (49,2048) ->fc1 -> hidden vector a (49,) word vector (num of lstm unit) ->fc2-> hideen vector b (49,) attention = softmax(hidden vector a + hidden vector b)

I don't think the attention could work, because:

1. hidden vector a is constant as image context never change
2. hidden vector b is generate from lstm output, which shouldn't contain location information

So in the end, fc2 will generate very small hidden vector b, and attention is mainly depend on the hidden vector a. That's why you observe attention maps are pretty similar across token.