A Simple Neural Network Module For Relational Reasoning

[howardyclo]

Metadata

• Authors: Adam Santoro, David Raposo, (+4 authors), Timothy P. Lillicrap
• Organization: DeepMind
• Conference: NIPS 2017
• Publish Date: 2017.06

[howardyclo]

Summary

This paper purposes a Relation Networks (RN) to solve relational problems, and achieves SOTA on VQA challenge on CLEVER dataset, text-based QA on bAbI dataset, and complex reasoning about dynamic physical systems. They also show that CNN do not have general capacity to solve relational questions but can gain this capacity when augmented with RN.

Related to #16. (New SOTA)

Relation Networks

• RN(O) = fΦ ( Σ{i,j} g_θ (o_i, o_j) )
• O = {o_1, o_2, ... o_n} (a set of objects). f_Φ and g_θ is MLP.
• The RN functional form considers the potential relations between all object pairs.
• The RN functional form ensures that it is invariant to the order of objects in the input.

Dealing with Unstructured Inputs

• In RN's simplest form, it operates on objects and hence do not explicitly operates on images or natural language. But, they can be encoded by CNN or LSTM and view output embeddings as objects.
• Conditioning RNs with question embedding: Just feed it to g_θ as additional input.
• In text-based QA (bAbI), view each sentence embedding as object.

Hyperparameters

• CLEVER
  • 4 conv layers with each 24 kernels, ReLU, batch norm. (ResNet or VGG).
  • LSTM for question embeddings with hidden size 128 . Word embedding with hidden size 32.
  • g_θ: 4-layer MLP with hidden size 256, ReLU per layer.
  • f_Φ: 3-layer MLP with each hidden size 256, 256 (dropout 0.5), 29, ReLU.
  • Adam optimizer with 2.5e-4.
  • Batch size 64 and distributed training with 10 workers.
• The hyperparameters for other tasks are similar.

Results

See the rest results in the paper.