Relational recurrent neural networks

[flrngel]

https://arxiv.org/abs/1806.01822 Paper from Deepmind

Abstract

• paper's model built with intuition
• so it is unclear why this works well
• shows state-of-the-art on the WikiText103, Project Gutenberg, and GigaWord datasets

1. Introduction

• toy tested to stress current models; LSTMs, DNC (from differentiable neural computers)
• Relational Memory Core(RMC) uses
  • multi-head dot product attention (see Transformer)
  • memory interactions

2. Relational reasoning

• this is process of understanding the ways in which entities are connected
• author claims that current models can be cast as relational reasoning
  • convolutional can be said to compute a relation of pixels (entities)
  • node and edge in message passing neural networks are entities
  • learnable nodes-edge connection can be relation
  • see also [13, 14, 17]
• but in current model, it is unclear whether inductive biases makes limitation
  • memory augmented networks

3. Model

• paper's model is assembled with
  1. LSTMs
  2. memory augmented
  3. non-local net
• paper's model applied attention between memories at a single time step
  • this differs from current models that uses attention across all previous representations

3.1. Allowing memories to interact using MHDPA

• should note that MHDPA uses KQV linears equals to which means proposed update to M(memory)

3.2. Encoding new memories

3.3. Introducing recurrence and embedding into an LSTM

see also Official tensorflow implemenation

means row/memory-wise MLP with layer normalization

4. Experiments

• they tested
  • 4.1. Illustrative supervised tasks
  • N-th farthest
  • Program Evaluation
  • 4.2. Reinforcement learning
  • Mini Pacman with viewport
  • 4.3. Language Modeling
  • WikiText103
  • Project Gutenberg
  • GigaWord datasets

5. Results

Figure3 is from n-th farthest task

6. Discussion

• author proposed inutions for the mechanisms for complex relational reasoning
• author considers their result primarily as "evidence of improved function"

Appendix