About attention matrix

[liyucheng09]

We have two attention matrix in Class Selector

One is relation_matrix, the other one is attention_matrix. Both of them is a matrix with shape(num_class, hidden_states).

I have checked in OpenNRE tensorflow version, it has only one relation_matrix, not two.

So WHY dose it use to matrix to model attention? Thanks!

Code in Pytorch

    def _attention_train_logit(self, x):
        relation_query = self.relation_matrix(self.attention_query)
        attention = self.attention_matrix(self.attention_query)
        attention_logit = torch.sum(x * attention * relation_query, 1, True)
        return attention_logit

Code in tensorflow

def __logit__(x, rel_tot, var_scope=None):
    with tf.variable_scope(var_scope or 'logit', reuse=tf.AUTO_REUSE):
        relation_matrix = tf.get_variable('relation_matrix', shape=[rel_tot, x.shape[1]], dtype=tf.float32, initializer=tf.contrib.layers.xavier_initializer())
        bias = tf.get_variable('bias', shape=[rel_tot], dtype=tf.float32, initializer=tf.contrib.layers.xavier_initializer())
        logit = tf.matmul(x, tf.transpose(relation_matrix)) + bias
    return logit