Closed skydoorkai closed 5 years ago
maybe the mountain looks like this:
picture from how neural networks are trained
and according to the point of view (empirically) in this paper the loss surfaces of multilayer networks:
For large-size networks, most local minima are equivalent and yield similar performance on a test set.
The probability of finding a “bad” (high value) local minimum is non-zero for small-size networks and decreases quickly with network size.
Struggling to find the global minimum on the training set (as opposed to one of the many good local ones) is not useful in practice and may lead to overfitting.
这篇文章比较新一点,有一个结论是用skip connection可以使loss surface 更加 convex。这里有更多的例子:https://www.cs.umd.edu/~tomg/projects/landscapes/
This 2015 paper has already done the work of the model accuracy drawing along two model interpolation path , and get the similar result.
If we think model A and B with similar accuracy are on the same mountain, if must satisfy: model M = A + alpha * (B - A) for alpha between (0, 1), the accuracy of M should not be significantly lower than the accuracy of A or B. So, from the above drawing, the models are on different mountains.
From the paper Visualizing the Loss Landscape of Neural Nets, it said (1)if a model architecture is good enough (such as Resnet), the landscape is nearly convex, so training can converge. If a model architecture is bad(such as Resnet without skip connections), there are many local minima of the loss, and the training will fail. (2) This 2015 paper is not a good way to compare two models with values along, as there are many invariance in the model, such as scale invariance from BatchNorm, or more complicated invariance from Relu (the negative input part). So even if the values of layer diff a lot between two models, they may be equivalent. It propose a filter normalization to solve the scale invariance problem.
There must be lots of invariance for a model, and each invariance will result in infinite mountains with some height, but significant difference in model values.
Also, there are some research papers stating that there are no spurious local minima in some DL models, so each local minima is some as global minima. But this paper Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima gives out a sample that there exists a spurious local minima, but state that if the initialization is good (using weight normalization on random values), the probability of avoiding the spurious local minima is very high. So spurious local minima is not a problem.
More experiments
We will generate some 2D heatmaps from 3 models to represent the model accuracy in the model space on a cross section defined by the 3 models.
Giving model A, B, C, we define the models on the cross section as: M(u, v) = A + u (B - A) + v (C - A) so M(0, 0) = A, M(1, 0) = B, M(0, 1) = C Here, we use two vectors (B - A) and (C - A) to create a 2D space. Note that (B - A) and (C - A) are not orthogonal, though we draw the 2D heatmap as if u and v axis are orthogonal to simplify the drawing. We draw a patch with u and v in (-0.5, 1.5) range. In the drawing title, Best is the best accuracy in this 2D patch.
A, B, C training with different random initializations
A, B, C are on different mountains, though their accuracies are nearly the same.
A, B, C training with same random initialization, but with different training data order
Also, A, B, C are on different mountains, though the ravines between them are not deep as 1.
A, B, C training with same initialization from a model with accuracy 0.339, but with different training data order
(a) after 0.2 epoch
For the area inside of A, B, C, all the models on it have similar or even a little higher accuracy than A, B, C. So, A, B, C are on the same mountain.
(b) after 1 epoch Still similar as (a)
(c) after 9 epoch
Now, though the the area inside A, B, C still has good accuracy, A, B, C are the highest.
A, B, C are still in the same mountain, but on different high rocks.
Another experiment
Instead of training for high accuracy to find the highest mountain, we train the model to get the lowest accuracy to find the abyss by changing the loss function from:
loss_fn = torch.nn.CrossEntropyLoss()
loss = loss_fn(output, target)
to:
softmax_fn = torch.nn.Softmax()
nllloss_fn = torch.nn.NLLLoss()
softmax_out = softmax_fn(output)
softmax_out = torch.add(softmax_out, -1.0)
log_out = torch.log(-softmax_out)
loss = nllloss_fn(log_out, target)
A, B, C are trained with different random initialization, with different training data order.
Also, accuracy of the models from linear interpolation along A, and B as M(x) = A + x * (B - A), with x in (-1, 2) range.
From the experiments above, we can see that for CIFAR10 with Resnet18, any random initialization will result in the vast plain with accuracy = 0.1.
If starting from the plain(acc=0.1), with the randomness in the training data, each worker will climb to different mountains.
If the staring point of all the workers are already on a mountain (such as at acc = 0.339), they will climb on the same mountain, with similar pace (the accuracy is determined on the number of training data it processed until it can climb no further higher).
There are infinite number of mountains, and also there are infinite number of abysses. From the experiments, all the mountains from Resnet18 training have similar height. Probably that Resnet18 is a good model, so there is no spurious local minima.
Experiment Setup
dataset: CIFAR10 model: resnet18 Init the model with random values, train for 30 epoch, save the model twice, at epoch 0 (just after initialization), and at epoch 30. Running the experiment for 8 times with different initial values, we get 8 models (A, B, C, D, E, F, G, H). Each model has two versions (just after init, and at epoch 30). For example, A = (A0, A30).
Model Accuracy at epoch 30
All the models reach similar accuracy around 91%.
Distance Among Models
Euclidean distance is used for measuring the distance between two models. At epoch 0, we get the distances among these 8 models, they are all around 63.
At epoch 30, they are still distantly apart, around 43.
Also, the distance between epoch 0 and epoch 30 for each of the models.
We think the training process as finding the highest mountain in the high-dimensional model space, with the model parameters as the dimensions in this high-dimensional space. From the distance values above, the 8 models are climbing different mountains, and these mountains have similar altitude. The reason for the existing of multiple (or even infinite) highest mountains is that there may exist multiple (or even infinite) solutions to the optimization problem in the model training.
Accuracy Along Model Linear Interpolation Path
Giving 2 models M1 and M2, linear interpolation is used to get the model between these two in the model space:
The accuracies along 7 paths between model A and other 7 models at epoch 30 are depicted below.![m8](https://user-images.githubusercontent.com/16055370/51318473-63e65700-1a95-11e9-95b5-7bce75c78c62.png)
This verifies our assumption that each model training is climbing a different mountains.
Conclusion
There is no mountain in the model space that is higher than any other mountains.
There are numerous (or infinite) mountains, all with the highest altitude. Initial value and training strategy will determine which mountain a training process will eventually climb.