Add callouts on expected questions

[bpmweel]

For each episode we should add callouts on expected questions.

Some questions we will want to postpone to a later point in the lesson. Others will often come up and can be answered immediately.

@florian-huber can you add a list of questions we had in our workshop to this issue?

• [ ] Size of training data needed
• [ ] How to decide on the number of layers/nodes?
• [ ] How do I search for pre-trained models or existing architectures?
• [ ] Explainable AI: How do you know what the neural network is responding to?
• [ ] Why do we use "relu" as activation function?
• [ ] Is Dropout (random silencing of nodes) applied per epoch or per batch?
• [ ] Is it necessary to remove co-linear (or highly correlated) features?

[florian-huber]

Questions we got (and handled) as Q&A in the collaborative documents:

• What is sparse deep learning? Only one example available?

  • Sparse deep learning is a recent variety where the number of parameters in a deep neural network is greatly reduced. Without harming performance. So the sparsity is in the parameters (i.e. bias and weights) in the network, not in the input data. See this blog.

• How do I search for pre-trained models or existing architectures?

  • You can google for your problem, or use so called ‘model zoos’, such as https://modelzoo.co/. Keras has a lot o pretrained models easily available: https://keras.io/api/applications/

• Why is this training code faster

  history = model.fit(X_train, y_train, epochs=100, use_multiprocessing=False)

  then this one?

  history = model.fit(X_train, y_train, epochs=100, use_multiprocessing=True)

  • Hard to say, concurrency in Python is a tricky subject. Multiprocessing has a lot of communication overhead. Without that argument threading is used. You can see on your system monitor all the cores are used. TensorFlow is taking care of parallelizing your problem in the most efficient way. Probably this argument is left for compatibility only.

• Is it possible to get feature importance?

  • Yes, but it is a bit more advanced, see https://christophm.github.io/interpretable-ml-book/feature-importance.html or https://towardsdatascience.com/feature-importance-with-neural-network-346eb6205743

We also received many good questions during the lessons that we responded to right away:

• How to decide on the number of layers/nodes? (or similar questions on the architecture choice)
  • Explaining how to choose the right DL network architecture is really difficult. Mostly, because there are no clear rules! A lot comes down to experience (and/or looking for what other people did for similar problems) and then trial & error...
• Why do we use "relu" as activation function?
• Is the choice of the activation function for the hidden layers and the output layer connected?
• What is an epoch?
• Is Dropout (random silencing of nodes) applied per epoch or per batch?
  • Neither: It is actually applied per sample!
• Is it necessary to remove co-linear (or highly correlated) features?
• Why does Batchnorm use xxx- parameters (4 per node)?
• How to avoid getting stuck in a local minimum? (or how does Adam/SGD avoid that)

[dafnevk]

Another question that often pops up: How many training data is needed for deep learning?