Question: How does back propagation loop?

[jinzishuai]

Specially, there are two iterations loops

• Loop over all sample data
• Iteration of the optimization algorithm like gradient descent or any other advanced optimization function such as @fminunc

Which is the inner loop and which is the outer loop?

[jinzishuai]

My Answer

• The outer loop is the training loop or iteration of the optimization algorithm like gradient descent or any other advanced optimization function such as @fminunc
• The inner loop is over the sample data, to compute the value and gradient of lost function. In fact, this loop can be implemented by Matrix operations (so that it might not look like a loop)