Deep Learning ideas

[dkmiller]

Read the papers in Deep Learning’s Most Important Ideas.

• [x] Tackling ImageNet with AlexNet and Dropout
  • [x] ImageNet Classification with Deep Convolutional Neural Networks
  • [x] Improving neural networks by preventing co-adaptation of feature detectors
  • [x] One weird trick for parallelizing convolutional neural networks
• [ ] Playing Atari with Deep Reinforcement Learning
• [ ] Encoder-Decoder Networks with Attention
  • [ ] Sequence to Sequence Learning with Neural Network
  • [ ] Neural machine translation by jointly learning to align and translate
• [ ] Adam Optimizer
• [ ] Generative Adversarial Networks (GANs)
• [ ] Residual Networks (ResNet)
• [ ] Transformers
• [ ] BERT and fine-tuned NLP Models
  • [ ] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  • [ ] The Annotated Transformer

---

• [ ] Graph-based, self-supervised program repair from diagnostic feedback

[dkmiller]

Learnings:

• https://registry.opendata.aws/
• https://www.kaggle.com/datasets
  • :warning: hard to download programmatically
• A database of open databases?
• Randomly grab subsets of images when training.
• Data parallelism for CNNs, model parallelism for fully-connected NNs.

To-dos:

• [ ] https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html