how to deal with overfitting?

[udibr]

It would be nice if the README will contain some tips on how to detect and avoid overfitting. I'm running the system and I believe I am overfitting.

So here are my tips which need editing before I put them in any README because I'm sure I dont understand the entire picture: 1) Currently I'm running on 50K examples so perhaps this is the main cause of the problem. 2) Also the output from running looks like the dump below. If I understand correctly the last two numbers are supposed to both fall as the iterations progress but as you can see they just wounder around. Maybe this is another sign of overfitting. I'm guessing that the first 3 numbers are distance to nearst-neighbours on different sample sizes. But is 55-53 a low or high number? If it is a low number then this is yet another indication of overfitting.

0 55.06 53.97 53.16 4.0025 0.2856
1 58.91 57.44 56.40 4.7697 0.7055
2 57.16 54.62 52.88 1.7402 0.4073
3 58.61 55.97 54.14 2.1908 0.3683
4 57.77 54.55 52.74 2.6172 0.3062
5 53.55 51.07 49.17 3.7945 0.1111
6 56.28 52.57 50.30 5.4140 0.1525
7 57.81 53.84 51.49 5.6486 0.1883
8 56.94 53.39 51.10 6.3922 0.0688
9 59.04 55.49 53.08 3.5038 0.3072
10 55.08 51.79 49.73 4.6309 0.0904
11 56.03 52.80 50.83 3.6019 0.1094
12 55.67 52.30 50.22 5.2213 0.3286
13 55.65 52.55 50.65 4.2390 0.2232

[Newmu]

etl_test.png is some debug code used it to check scaling/transpose bugs with image data formats - it shows you actual training examples - those are not samples.

The last number is cost of the discriminator, the second to last is cost of generator. A typical failure mode is the generator eventually being unable to fool the discriminator resulting in its cost climbing to 10-15-20 and the discriminator cost going to 0.

The first three are nearest neighbor euclidean distances of real examples to 1,000, 10,000, 100,000 generated samples. One way of measuring overfitting is whether that number is lower on training data than on validation data. It's a relative number as it's dataset dependent. Ideally it will continue to go down over the course of training.

[udibr]

Thanks, I've spotted my mistake with etl_test and the real generated images looks more resonable. So it looks like both networks are matching each other, and the cost numbers it looks like no progress is being made. However, if I look at the sequence of the generated images after each iteration there is a very clear improvement from iteration to iteration. So what is going on? Anyway its amazing!

[udibr]

It now looks that what you described is happening the generator cost starts to rise and the discriminator falls dramatically. Any tips on how to rebalance this? (I am collecting more images, in the hope that this will help.)

[soumith]

@udibr some tips on re-balancing are given here: http://torch.ch/blog/2015/11/13/gan.html#balancing-the-gan-game

[Newmu]

We're still figuring out/understanding why this happens. Soumith's link has good advice. More data helps but does not prevent it - we've seen models go unstable before even finishing an epoch.

[udibr]

Thanks, the link is great. My problem is oscilliaitions between the two scores and not an explosition of one on the expense of the other. The image quality just get better after each iteration but this could be overfitting in action. I will re-run with more images and see if it helps.

[dribnet]

Hi @udibr - one thing to be on the lookout for in the faces code is the expected fuel dataset. The faces code expects data stored channel last like image[w][h][c] instead of the fuel standard channel first pattern of image[c][w][h].

The etl_test.png is a great test of whether this is is being unpacked correctly - as @Newmu says it's a logical passthrough of training samples and should not be garbled at all. If it is, there is a one line numpy.dstack you can insert to transform your data appropriately.

[nguyenductung]

Hi, I'm trying your network with my dataset (~50K samples) and I also think it is overfitting. Dropout may help prevent overfitting but it seems that you do not use dropout in your training script for the faces dataset. There is a dropout function in lib/ops.py, could you tell me how to use it?

[udibr]

add it after every relu line. For example the discriminator will look like:

        h = lrelu(dnn_conv(X, w, subsample=(2, 2), border_mode=(2, 2)))
        h = dropout(h, 0.2)
        h2 = lrelu(batchnorm(dnn_conv(h, w2, subsample=(2, 2), border_mode=(2, 2)), g=g2, b=b2))
        h2 = dropout(h2, 0.2)
        h3 = lrelu(batchnorm(dnn_conv(h2, w3, subsample=(2, 2), border_mode=(2, 2)), g=g3, b=b3))
        h3 = dropout(h3, 0.2)
        h4 = lrelu(batchnorm(dnn_conv(h3, w4, subsample=(2, 2), border_mode=(2, 2)), g=g4, b=b4))
        h4 = dropout(h4, 0.2)
        h4 = T.flatten(h4, 2)
        y = sigmoid(T.dot(h4, wy))

[nguyenductung]

@udibr Thanks. I'm trying it now.