Introduction to Deep Learning

The repository latex-math is used. Please read its ReadMe here: https://github.com/compstat-lmu/latex-math

General rules

Please observe the following rules when creating and editing the lecture and exercise slides:

Setup

1. Clone this repository
2. Clone the latex-math repository into the main directory of this repository
3. Navigate to a folder where the slideset is contained, e.g. 2020/01-introduction
4. If there is Makefile in the folder: do make -f "Makefile" unless render slides by knitr::knit2pdf("slides.Rnw")

Structure

Topics will be added or remade, the normal ones are already in the slides:

1. Introduction, Overview, and a Brief History of Deep Learning

2. Deep Feed-Forward Neural Networks, Gradient Descent, Backpropagation, Hardware and Software

3. Regularization of Neural Networks, Early Stopping

4. Dropout and Challenges in Optimization

5. Advances in Optimization

6. Activation Functions and Initialization

7. Convolutional Neural Networks, Variants of CNNs, Applications

8. Modern CNNs and Overview of some Applications

9. Recurrent Neural Networks

10. Modern RNNs and Applications

11. Deep Unsupervised Learning

12. Autoencoders, AE Regularization and Variants

13. Manifold Learning

14. Deep Generative Models, VAE, GANs

Math and formula

1. Math environments within a text line are created by $ environment, separate equation lines are created by $$ environment

2. The abbreviations defined in the header file should always be used within the code for simplification purposes

3. The repo latex-math is used. Please read the corresponding ReadMe: https://github.com/compstat-lmu/latex-math

Material Deep Learning

Extra material to have a look at:

• Alex J. Smola (2020): Dive into Deep Learning (An interactive deep learning book with code, math, and discussions Provides NumPy/MXNet, PyTorch, and TensorFlow implementations)(free HTML version)
• Goodfellow, Bengio, Courville (2016): Deep Learning (free HTML version)
• Awesome Deep Learning
• Andrej Karpathy blog
• Coursera Kurs "Neural Networks for Machine Learning"
• Youtube Talk von Geoff Hinton "Recent Developments in Deep Learning"
• Practical Deep Learning For Coders - contains many detailed python notebooks on how to implement different DL architectures
• The Matrix Calculus You Need For Deep Learning
• Tensorflow Neural Network Playground
• A Weird Introduction to Deep Learning
• Deep Learning Achievements Over the Past Year
• Scalable Deep Learning (Talk)
• Deep Learning Resources

Good Websites to have a look

• distill.pub: in-depth explanations of important concepts, worth checking out periodically for new material

Optimization / Training of NNs:

• Why Momentum Really Works
• Adam -- latest trends in deep learning optimization
• Overview of Gradient Descent Optimization Algorithms
• Yes you should understand backprop
• A Recipe for Training Neural Networks

Regularization:

• Regularization for Deep Learning: A Taxonomy

CNNs:

• The Sobel and Laplacian Edge Detectors
• Keras Blog: How convolutional neural networks see the world
• The 9 Deep Learning Papers You Need To Know About
• Python based visualization repo for CNNs
• Computing Receptive Fields of Convolutional Neural Networks
• Deconvolution and Checkerboard Artifacts
• Understanding Convolution in Deep Learning
• [What do we learn from region based object detectors (Faster R-CNN, R-FCN, FPN)]
• How Convolutional Neural Networks see the World
• Attention in Neural Networks and How to Use It
• Neural Networks - A Systematic Introduction (FU Berlin)
• Deep Learning - The Straight Dope (contains notebooks designed to teach deep learning)
• Computing Receptive Fields of Convolutional Neural Networks
• Stanford: Convolutional Neural Networks for Visual Recognition
  • Videos
  • Assignements und notes

Autoencoders

• PCA
• Introducing Variational Autoencoders (in Prose and Code)

Variational Autoencoders

• A Tutorial on Variational Autoencoders

Reinforcement Learning

• Statistical Reinforcement Learning (Lecture)
• Practical Reinforcement Learning (Course)

LSTMs:

• Understanding LSTM and its diagrams
• The most comprehensive yet simple and fun RNN/LSTM tutorial on the Internet.

Hyperparameter Optimization / Neural Architecture Search / etc:

• Using Machine Learning to Explore Neural Network Architecture

Software / Languages / etch

• R vs Python: Image Classification with Keras
• H20 related stuff:
  • Repository that contains the H2O presentation for Trevor Hastie and Rob Tibshirani's Statistical Learning and Data Mining IV course in Washington, DC on October 19, 2016.
  • Deep Learning with H2O - PDF/Book

Nice Demos and Vizualisations

• Deep Traffic

Material for Exercises

• Neural networks Exercises (Part-1)

External material

1. other
   • PyData-2017-TF_TFS: Slides for Spark + Tensorflow + Notebooks