Attention Mechanisms and Memory Networks

[mac40]

• [x] Study
• [x] Implement RNN
  • [x] LSTM
  • [x] GRU
  • [ ] Autoencoders
• [x] Test
  • [x] activate/deactivate Attention
  • [x] dropout
  • [x] double vs single RNN layer
  • [x] visualize attention scores

Useful references:

• A Beginner's Guide to Attention Mechanisms and Memory Networks
• Intuitive Understanding of Attention Mechanism in Deep Learning
• Tutorial on Attention-based Models (Part 1)
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

[mac40]

Useful documentation for attention implementation:

• Attention in Deep Networks with Keras
• A neural attention model for speech command recognition
• Keras LSTM tutorial
• Understanding LSTM
• Deep Learning #4: Why You Need to Start Using Embedding Layers
• Speech Recognition with Neural Networks

documentation for attention visualization

• How to Visualize Your Recurrent Neural Network with Attention in Keras
• keras-attention repo
• How to visualize attention weights?

documentation for RNN autoencoders

• Step-by-step understanding LSTM Autoencoder layers
• How to Use the TimeDistributed Layer in Keras

[mac40]

Pausing issue before closure unless autoencoders automagically solve themselves